{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_Linear_Regression",
      "provenance": [],
      "collapsed_sections": [
        "sdruDHf_laWg"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7nDdR3N2R2PJ",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://practicalai.me\"><img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/rounded_logo.png\" width=\"100\" align=\"left\" hspace=\"20px\" vspace=\"20px\"></a>\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/basic_ml/04_Linear_Regression/linear_regression.png\" width=\"150\" vspace=\"20px\" align=\"right\">\n",
        "\n",
        "<div align=\"left\">\n",
        "<h1>Linear Regression</h1>\n",
        "\n",
        "In this lesson we will learn about linear regression. We will understand the basic math behind it, implement it in Python. and then look at ways of interpreting the linear model.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Izx6GIn_SETR",
        "colab_type": "text"
      },
      "source": [
        "<table align=\"center\">\n",
        "  <td>\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/rounded_logo.png\" width=\"25\"><a target=\"_blank\" href=\"https://practicalai.me\"> View on practicalAI</a>\n",
        "  </td>\n",
        "  <td>\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/colab_logo.png\" width=\"25\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/practicalAI/practicalAI/blob/master/notebooks/04_Linear_Regression.ipynb\"> Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/github_logo.png\" width=\"22\"><a target=\"_blank\" href=\"https://github.com/practicalAI/practicalAI/blob/master/notebooks/basic_ml/04_Linear_Regression.ipynb\"> View code on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "source": [
        "# Overview"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LaFUStk-24I4",
        "colab_type": "text"
      },
      "source": [
        "Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n",
        "\n",
        "$\\hat{y} = XW + b$\n",
        "* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n",
        "* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n",
        "* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n",
        "* $b$ = bias | $\\in \\mathbb{R}^{1}$ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QAgr7Grv9pb6",
        "colab_type": "text"
      },
      "source": [
        "* **Objective:**  Use inputs $X$ to predict the output $\\hat{y}$ using a linear model. The model will be a line of best fit that minimizes the distance between the predicted (model's output) and target (ground truth) values. Training data $(X, y)$ is used to train the model and learn the weights $W$ using gradient descent.\n",
        "* **Advantages:**\n",
        "  * Computationally simple.\n",
        "  * Highly interpretable.\n",
        "  * Can account for continuous and categorical features.\n",
        "* **Disadvantages:**\n",
        "  * The model will perform well only when the data is linearly separable (for classification).\n",
        "  * Usually not used for classification and only for regression.\n",
        "* **Miscellaneous:** You can also use linear regression for binary classification tasks where if the predicted continuous value is above a threshold, it belongs to a certain class. But we will cover better techniques for classification in future lessons and will focus on linear regression for continuous regression tasks only.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xP7XD24-09Io",
        "colab_type": "text"
      },
      "source": [
        "### Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "476yPgTM1BKJ",
        "colab_type": "text"
      },
      "source": [
        "1. Randomly initialize the model's weights $W$ (we'll cover more effective initialization strategies in future lessons).\n",
        "2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n",
        "  * $\\hat{y} = XW + b$\n",
        "3. Compare the predictions $\\hat{y}$ with the actual target values $y$ using the objective (cost) function to determine the loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it.\n",
        "\n",
        "  * $J(\\theta) = MSE = \\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2 $\n",
        "    * ${y}$ = ground truth | $\\in \\mathbb{R}^{NX1}$\n",
        "    * $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$\n",
        "4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights.\n",
        "  * $J(\\theta) = \\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2  = \\frac{1}{N}\\sum_i (y_i - X_iW)^2 $\n",
        "    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW) X_i = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i) X_i$\n",
        "    * $\\frac{\\partial{J}}{\\partial{b}} = -\\frac{2}{N} \\sum_i (y_i - X_iW)1 = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i)1$\n",
        "5. Update the weights $W$ using a small learning rate $\\alpha$. The simplified intuition is that the gradient tells you the direction for how to increase something so subtracting it will help you go the other way since we want to decrease loss $J(\\theta)$:\n",
        "  * $W = W - \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "  * $b = b - \\alpha\\frac{\\partial{J}}{\\partial{b}}$\n",
        "6. Repeat steps 2 - 5 to minimize the loss and train the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wqEglvPJTifp",
        "colab_type": "text"
      },
      "source": [
        "# Set up"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ebK53z_VctPM",
        "colab_type": "code",
        "outputId": "c8ebb86c-8df7-4465-957c-569c2595cbd1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Use TensorFlow 2.x\n",
        "%tensorflow_version 2.x"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HRXD7LqVJZ43",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import numpy as np\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NFsKg-Z6IWqG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Arguments\n",
        "SEED = 1234\n",
        "SHUFFLE = True\n",
        "NUM_SAMPLES = 50\n",
        "TRAIN_SIZE = 0.7\n",
        "VAL_SIZE = 0.15\n",
        "TEST_SIZE = 0.15\n",
        "NUM_EPOCHS = 100\n",
        "BATCH_SIZE = 10\n",
        "INPUT_DIM = 1\n",
        "HIDDEN_DIM = 1\n",
        "LEARNING_RATE = 1e-1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K8uaUq8XSsQ8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Set seed for reproducibility\n",
        "np.random.seed(SEED)\n",
        "tf.random.set_seed(SEED)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jvJKjkMeJP4Q",
        "colab_type": "text"
      },
      "source": [
        "# Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RuPl9qlSJTIY",
        "colab_type": "text"
      },
      "source": [
        "We're going to create some simple dummy data to apply linear regression on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w5T7yGfiEQnx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l_a--YWXdwig",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "McZ7wHsve61A",
        "colab_type": "text"
      },
      "source": [
        "We're going to create some dummy data to train our linear regression model on. We're going to create roughly linear data (`y = 3.5X + 10`) with some random noise so the points don't all align in a straight line. Our goal is to have the model converge to a similar linear equation (there will be slight variance since we added some noise)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jNWcn3uadzpJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Generate synthetic data\n",
        "def generate_data(num_samples):\n",
        "    \"\"\"Generate dummy data for linear regression.\"\"\"\n",
        "    X = np.array(range(num_samples))\n",
        "    random_noise = np.random.uniform(-10,20,size=num_samples)\n",
        "    y = 3.5*X + random_noise # add some noise\n",
        "    return X, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ppGTtKRrdwgF",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4f9KSJR6flBt",
        "colab_type": "text"
      },
      "source": [
        "Now let's load the data onto a dataframe and visualize it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2mb2SjSQIWvF",
        "colab_type": "code",
        "outputId": "5a96430d-6776-4356-d685-2ce7fd7ddf26",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Generate random (linear) data\n",
        "X, y = generate_data(num_samples=NUM_SAMPLES)\n",
        "data = np.vstack([X, y]).T\n",
        "df = pd.DataFrame(data, columns=['X', 'y'])\n",
        "X = df[['X']].values\n",
        "y = df[['y']].values\n",
        "df.head()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>X</th>\n",
              "      <th>y</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0</td>\n",
              "      <td>-4.254416</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1.0</td>\n",
              "      <td>12.163263</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>10.131832</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3.0</td>\n",
              "      <td>24.060758</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.0</td>\n",
              "      <td>27.399274</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     X          y\n",
              "0  0.0  -4.254416\n",
              "1  1.0  12.163263\n",
              "2  2.0  10.131832\n",
              "3  3.0  24.060758\n",
              "4  4.0  27.399274"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6LwVVOkiLfBN",
        "colab_type": "code",
        "outputId": "ed7e32c6-c0e2-447c-ad0c-dc4e4d734597",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "source": [
        "# Scatter plot\n",
        "plt.title(\"Generated data\")\n",
        "plt.scatter(x=df['X'], y=df['y'])\n",
        "plt.show()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAbGElEQVR4nO3de5RdZXnH8e/PSdAR0OEyxjAQEzSE\ngtSkTlEbtYgXorUSWS6EqsVKG2+spRXRoF2KdlHSUqV2YbGhUrBLEcolspClpoSKtaBOTMpFoAaE\nkjEkwyUFJY0kPP3j7IHN5JyZc/bZ57L3/n3WOmvOeffe57w7DM955708ryICMzMrl2f1ugJmZpY/\nB3czsxJycDczKyEHdzOzEnJwNzMrIQd3M7MScnA36wBJ75X0Hy2cf6+kN3SyTlYtDu7WNZJOkvQj\nSb+WtC15/iFJ6nXdppL075L+tNf1qEdSSHpJr+th/c3B3bpC0unAl4BzgRcCc4APAEuBvbpcl1nd\n/DyzXnBwt46T9Hzg88CHIuKKiHgsajZExLsiYmdy3rMl/a2k/5G0VdJXJA0mx46RtFnS6Umrf4uk\nP0l9RjPXflLSA8A/S9pP0rWSJiQ9kjw/ODn/bOA1wPmSfiXp/KT8cElrJT0s6S5JJ6Y+/wBJ10h6\nVNKPgRfP8G/yHkn3SXpI0qenHDta0k2Stif3eb6kvZJjNyan/VdSt3dOdy9WXQ7u1g2vAp4NfGuG\n81YBhwGLgZcAI8BnUsdfCDw/KT8V+LKk/Vq4dn/gRcAKar/7/5y8ngfsAM4HiIhPAz8ATouIfSLi\nNEl7A2uBbwAvAE4C/kHSEcn7fxn4P2Au8L7kUVdyzQXAe4CDgAOAdDDeDfw5cCC1f7vXAx9K6vba\n5JyXJXW7bLp7sQqLCD/86OgDeDfwwJSy/wS2UwtErwUE/Bp4ceqcVwG/SJ4fk5w7K3V8G/DKJq/9\nDfCcaeq4GHgk9frfgT9NvX4n8IMp1/wj8FlgAHgCODx17K+A/2jwWZ8Bvpl6vXdSvzc0OP+jwNWp\n1wG8pNl78aOaD/c9Wjc8BBwoaVZE7AKIiN8DkLSZWstzGHgusD41vipqgfOp95m8PvE4sE+T105E\nxP89dVB6LnAesAyYbP3vK2kgInbXuYcXAa+QtD1VNgv4l+TzZwH3p47dV/+fAqi11p86NyJ+Lemh\nVN0OA74IjCb3NQtY3+jNMtyLVYC7ZawbbgJ2AsdPc86D1FrmR0bEUPJ4fkTs08T7N3Pt1PSnpwOL\ngFdExPOo/fUAtS+FeuffD3w/9f5DUesW+SAwAewCDkmdP2+a+m5Jn5sE5wNSxy8A7gQWJnX7VKpe\n9cx0L1ZBDu7WcRGxHfgctT7qd0jaV9KzJC2m1iVBRDwJXAicJ+kFAJJGJB3XxPtnuXZfal8I2yXt\nT617JW0rcGjq9bXAYclA6Ozk8buSfitpHV8FnCXpuUmf+inTfPYVwFslvToZKP08z/x/cV/gUeBX\nkg4HPjhD3Wa6F6sgB3frioj4G+BjwCeoBaet1PqsP0mt/53k+SbgZkmPAv9GrUXajFav/TtgkFqr\n/2bgO1OOfwl4RzL75O8j4jHgTdQGUn8JPAD8NbWBYoDTqHURPQBcTG2As66IuB34MLXB2S3AI8Dm\n1CkfB/4IeIzal9ZlU97iLOCSZDbNiU3ci1WQIrxZh5lZ2bjlbmZWQg7uZmYl5OBuZlZCDu5mZiXU\nF4uYDjzwwJg/f36vq2FmVijr169/MCKG6x3ri+A+f/58xsbGel0NM7NCkdRwJbS7ZczMSsjB3cys\nhBzczcxKyMHdzKyEHNzNzEqoL2bLmJlVzZoN45z73bv45fYdHDQ0yBnHLWL5kpHc3t/B3cysy9Zs\nGOfMq25lxxO1vVTGt+/gzKtuBcgtwM/YLSPpomRD4ttSZZdJ2pg87pW0MSmfL2lH6thXcqmlmVmJ\nnPvdu54K7JN2PLGbc797V26f0UzL/WJqm+1+bbIgIt45+VzSF4D/TZ1/d0QszquCZmZl88vtO1oq\nz2LGlntE3Ag8XO+YahtWnghcmluNzMxK7qChwZbKs2h3tsxrgK0R8fNU2QJJGyR9X9JrGl0oaYWk\nMUljExMTbVbDzKw4zjhuEYOzB55RNjh7gDOOa3bjsZm1O6B6Ms9stW8B5kXEQ5JeDqyRdGREPDr1\nwohYDawGGB0d9XZQZlZYrc58mTzWl7NlJM0CTgBePlkWETup7XJPRKyXdDdwGOCsYGZWSllnvixf\nMpJrMJ+qnW6ZNwB3RsRTG/tKGpY0kDw/FFgI3NNeFc3M+lc3Zr5k0cxUyEuBm4BFkjZLOjU5dBJ7\nDqS+FrglmRp5BfCBiKg7GGtmVgbdmPmSxYzdMhFxcoPy99YpuxK4sv1qmZkVw0FDg4zXCeSTM186\nvRK1EeeWMTNrw3QzXyb748e37yB4uj9+zYbxjtfLwd3MrA3Ll4xwzglHMTI0iICRoUHOOeEoli8Z\n6Wl/vHPLmJm1qdHMl172x7vlbmbWId1YidqIg7uZ2RRrNoyzdNU6Fqz8NktXrcvcR96NlaiNuFvG\nzCqr3kwWILd0vN1YidqIInq/8n90dDTGxryI1cy6Z+rKUqi1qp8z+1k88vgTe5w/MjTID1ce280q\nzkjS+ogYrXfMLXczK716LfRGM1mmlk3q9aKkVjm4m1mpNcr90iiIN9KNQdA8eUDVzEqtUQt9QKp7\n/tDg7J4NgubJLXczK7VG3Sm7IxicPbBHn/tZbzsS6M0gaJ4c3M2s1BrlfhlJ9b3XC+JFC+ZTObib\nWamdcdyiurNiJgN50YN4Iw7uZlZqvZxr3ksO7mbWU91IiVvmFnojDu5m1jNZt6izmTm4m1nPTJcS\ntx+De6823sjCwd3MeqZft6irp2h/ZTSzh+pFkrZJui1VdpakcUkbk8dbUsfOlLRJ0l2SjutUxc2s\n+HqZErdV/boRdiPNrFC9GFhWp/y8iFicPK4DkHQEtY2zj0yu+QdJA3WuNTPraUrcVhXprwxoIrhH\nxI3Aw02+3/HANyNiZ0T8AtgEHN1G/cysxKbboq7fFOmvDGivz/00SX8MjAGnR8QjwAhwc+qczUnZ\nHiStAFYAzJs3r41qmFm/yDLgWJRpitMthupHWROHXQC8GFgMbAG+0OobRMTqiBiNiNHh4eGM1TCz\nfjE54Di+fQfB0wOOWXcx6jdF+isDMrbcI2Lr5HNJFwLXJi/HgUNSpx6clJlZyRVtWmMWRfkrAzK2\n3CXNTb18OzA5k+Ya4CRJz5a0AFgI/Li9KppZERRtwLHsZmy5S7oUOAY4UNJm4LPAMZIWAwHcC7wf\nICJul3Q58DNgF/DhiGgtI76ZFVKj7Iv9OuBYdjMG94g4uU7xV6c5/2zg7HYqZWbFU7QBx7LzClUz\ny0VVsy/2Kwd3M8tNkQYcy857qJqZlZBb7mZWKEXKzNhLDu5mVhjTZWYE9/enObibWWE0Wih11jW3\ns3PXk4VJx9sNDu5m1hV5dKc0WhC1fccTe5SVbXVsqzygamYdl1femVYXRFV5dayDu5l1XF4bXTTK\n/77fc2fXPb/Kq2PdLWNmDeU1MyWvvDONFkoBXh07hYO7mdWV556heeadmW6hlGfLPM3B3czqyjOF\nb5a8M63+1eDVsc/k4G5mdeWZwrfVvDN5/tVQVQ7uZla3lZx3Ct9WWtZV2Pij0zxbxqziGk1TfN3h\nw3VnpnRjkNIbf7TPwd2s4hq1km+4c6Jne4Y2+uugylMbW+VuGbOKm66V3KtBSm/80T4Hd7OSaXWW\nST9uj+eNP9rXzB6qFwFvBbZFxEuTsnOBPwR+A9wN/ElEbJc0H7gDmFx2dnNEfKAD9TazOrLMMunX\nVrKnNranmT73i4FlU8rWAi+NiN8G/hs4M3Xs7ohYnDwc2M26KMsy/+VLRnrWt26d08wG2TcmLfJ0\n2fdSL28G3pFvtcwsi6yzTNxKLp88+tzfB1yWer1A0gbgUeAvIuIHOXyGmTWhW/3n3g2p/7U1FVLS\np4FdwNeToi3AvIhYAnwM+Iak5zW4doWkMUljExMT7VTDzBKNsibm2X+eV/pe66zMwV3Se6kNtL4r\nIgIgInZGxEPJ8/XUBlsPq3d9RKyOiNGIGB0eHs5aDbPSW7NhnKWr1rFg5bdZumrdtEG0G/3neaXv\ntc7K1C0jaRnwCeD3I+LxVPkw8HBE7JZ0KLAQuCeXmppVUJbZL53uP/fq0WKYseUu6VLgJmCRpM2S\nTgXOB/YF1kraKOkryemvBW6RtBG4AvhARDzcobqblV4/tpK9erQYmpktc3Kd4q82OPdK4Mp2K2Vm\nNf3YSu7XefH2TM4tY9bH+rGV7HnxxeD0A2Z9rF9byZ4X3/8c3M36mHOsWFYO7mZ9zq1ky8J97mZm\nJeTgbmZWQu6WMSso53ex6Ti4mxVQlpWrVi0O7mYFNN3K1emCu1v71eHgblZAWVauurVfLQ7uZh2Q\npYXcyjVZ8rZnbe1bMXm2jFnOsuQ7b/WaLHnb+zFPjXWOg7tZzrJkcmz1miz5XfoxT411jrtlzHKW\npYWc5ZpWV672a54a6wy33M1ylqWF3I1WtbM5Votb7mY5y9JC7lar2nlqqsPB3SxnWTI5Ovuj5U3J\n3tY9NTo6GmNjY72uhpkX+VihSFofEaP1jrnlbpbwIh8rk6YGVCVdJGmbpNtSZftLWivp58nP/ZJy\nSfp7SZsk3SLpdzpVebM89eNm1GZZNTtb5mJg2ZSylcD1EbEQuD55DfBmYGHyWAFc0H41zTrPi3ys\nTJoK7hFxI/DwlOLjgUuS55cAy1PlX4uam4EhSXPzqKxZJ3mRj5VJO33ucyJiS/L8AWBO8nwEuD91\n3uakbEuqDEkrqLXsmTdvXhvVMMvHdNMR8xpo9YCtdUsuA6oREZJamnYTEauB1VCbLZNHPcza0Wg6\nIpDLQKsHbK2b2gnuWyXNjYgtSbfLtqR8HDgkdd7BSZlZ36u3yGfpqnW5ZFN0VkbrpnbSD1wDnJI8\nPwX4Vqr8j5NZM68E/jfVfWNWOHkNtHrA1rqp2amQlwI3AYskbZZ0KrAKeKOknwNvSF4DXAfcA2wC\nLgQ+lHutzboor4FWD9haNzXVLRMRJzc49Po65wbw4XYqZdZP8hpodVZG6yavUDWbQV4Drc4fY93k\n3DJmGS1dta7uVncjQ4P8cOWxPaiRVc10uWWcz90sIw+QWj9zcDfLyAOk1s8c3M0yyrJJtVm3eEDV\nLCMPkFo/c3A3a4O3rbN+5W4ZM7MScnA3MyshB3czsxJyn7v1Hec8N2ufg7v1lSw5z1v9MvCXh1WB\nu2Wsr7S6SfXkl8H49h0ET38ZrNlQfwuBVs83KyoHd+srrS7pb/XLoNXzzYrKwd36SqtL+lv9MnA+\nGKsKB3frK60u6W/1y8D5YKwqHNytryxfMsI5JxzFyNAgopY+95wTjpp2A4xWvgycD8aqwrNlrCem\nm7HSypL+VvO7OB+MVUXmzTokLQIuSxUdCnwGGAL+DJhIyj8VEddN917erKNapk53hFrreboWupnt\nqSObdUTEXRGxOCIWAy8HHgeuTg6fN3lspsBu1eMZK2adl1ef++uBuyPivpzez0rMM1bMOi+v4H4S\ncGnq9WmSbpF0kaT96l0gaYWkMUljExMT9U6xkvKMFbPOazu4S9oLeBvwr0nRBcCLgcXAFuAL9a6L\niNURMRoRo8PDw+1WwwrEM1bMOi+P2TJvBn4aEVsBJn8CSLoQuDaHz7ASyTpjxTlhzJqXR3A/mVSX\njKS5EbElefl24LYcPsNKptUdjLIkFDOrsraCu6S9gTcC708V/42kxUAA9045ZpbJdDNs8soWaVYm\nbQX3iPg1cMCUsve0VSOzOlqdYeOWvlWd0w9YIbQ6w8Zz6a3qHNytEFqdYeO59FZ1Du5WCK0mFPNc\neqs6Jw6zwmhlhs0Zxy2qm7/Gc+mtKhzcrZSc/dGqzsHdSqvVufRmZeLgbrnwnHKz/uLgbm3znHKz\n/uPgbi2p10LPsnrUzDrLwd2a1qiFPjWwT/KccrPe8Tx3a1qjFvqAVPd8zyk36x233Cus1UHQRi3x\n3REMzh7wnHKzPuKWe0VNdrGMb99B8HQXy5oN4w2vadQSn1wt2uzqUTPrPLfcKyrLIOh0qz49p9ys\nvzi4V1SWxFpe9WlWHA7uFXXQ0CDjdQL5TIOgbqGbFYODewXUGzh1Yi2zcvOAask1GjgFPAhqVmJt\nt9wl3Qs8BuwGdkXEqKT9gcuA+dT2UT0xIh5p97OsddMNnP5w5bHef9SspPLqlnldRDyYer0SuD4i\nVklambz+ZE6fZS3IMnCaZ64Yf0mY9UanumWOBy5Jnl8CLO/Q59gMsuxIlNf+o1nm0ptZPvII7gF8\nT9J6SSuSsjkRsSV5/gAwJ4fPqZQ1G8ZZumodC1Z+m6Wr1mUOiK3uPQr57T/qTarNeiePbplXR8S4\npBcAayXdmT4YESEppl6UfBGsAJg3b14O1Simet0WQG7dIlnmpmedJjmVN6k26522g3tEjCc/t0m6\nGjga2CppbkRskTQX2FbnutXAaoDR0dE9gn8VNOrbfs7sZ+WaQrfVuel5TZPM60vCzFrXVreMpL0l\n7Tv5HHgTcBtwDXBKctopwLfa+ZyyatRt8cjjT9Q9v1st3uVLRnKZJpmlS8jM8tFuy30OcLVqKV9n\nAd+IiO9I+glwuaRTgfuAE9v8nFJqNVh3s8Wbx0pUpysw6522gntE3AO8rE75Q8Dr23nvKmjUbTE0\nOJudu55suVukH6cdOl2BWW94hWoPNeq2OOttR7bcLeJph2aW5twyPTRTt0UrLV7vY2pmaQ7uPZZX\nt4WnHZpZmrtlSiLLSlQzKy8H9y7Ja8VpI552aGZp7pbpgjwTcTXiaYdmlubg3gXdGuz0tEMzm+Tg\n3gV5D3b243x2M+sv7nPvgjwHOz2f3cya4eDeBXkOdjqNrpk1w90yXZDnYKfns5tZMxzcuySvwU6n\n0TWzZrhbpmA8n93MmuGWe8F4PruZNcPBvYA8n93MZuJuGTOzEnLLfRpeLGRmReXg3kA38sGYmXWK\nu2Ua8GIhMyuyzMFd0iGSbpD0M0m3S/pIUn6WpHFJG5PHW/Krbvd4sZCZFVk73TK7gNMj4qeS9gXW\nS1qbHDsvIv62/ep1R72+dS8WMrMiy9xyj4gtEfHT5PljwB1A4TqjGyXiet3hw14sZGaFlUufu6T5\nwBLgR0nRaZJukXSRpP0aXLNC0piksYmJiTyqkUmjvvUb7pzgnBOOYmRoEAEjQ4Occ8JRHkw1s0JQ\nRLT3BtI+wPeBsyPiKklzgAeBAP4SmBsR75vuPUZHR2NsbKytemS1YOW3qfcvIOAXq/6g29UxM2ua\npPURMVrvWFstd0mzgSuBr0fEVQARsTUidkfEk8CFwNHtfEaneWNpMyujdmbLCPgqcEdEfDFVPjd1\n2tuB27JXr/PyTsTV6Y2wzcya0c5smaXAe4BbJW1Myj4FnCxpMbVumXuB97dVww7LMxGXFz6ZWb9o\nu889D73sc8/T0lXr6k6fHBka5Icrj+1BjcyszDrW527P5IVPZtYvnFsmIy98MrN+5pZ7Bl74ZGb9\nzsE9Ay98MrN+526ZDKbrW/cuSWbWD0oZ3Du9yYb71s2s35WuW6ZRf3iei4nyXvhkZpa30rXcZ9pk\nI48WfZ4Ln8zMOqF0i5gaJQKDWus6HfgHZw94wNPMCqtSi5ga9XsPSN42z8wqo3TBvVF/+O4Gf6F4\n9aiZlVHpgvvyJSN155qPOLWvmVVI6QZUgYZzzdMZG8EzXMysvEoZ3OvxDBczq5LKBHdo3KI3Myub\n0vW5m5mZg7uZWSk5uJuZlVDHgrukZZLukrRJ0spOfY6Zme2pI8Fd0gDwZeDNwBHUNs0+ohOfZWZm\ne+pUy/1oYFNE3BMRvwG+CRzfoc8yM7MpOhXcR4D7U683J2VPkbRC0piksYmJiQ5Vw8ysmno2oBoR\nqyNiNCJGh4eHe1UNM7NS6tQipnHgkNTrg5OyXHV6xyUzs6LqVHD/CbBQ0gJqQf0k4I/y/IDJHZcm\nc8VM7rgEOMCbWeV1pFsmInYBpwHfBe4ALo+I2/P8jJl2XDIzq7KO5ZaJiOuA6zr1/o3ysDs/u5lZ\ngVeoNsrD7vzsZmYFDu6NdlxyfnYzswKn/HV+djOzxgob3MH52c3MGilst4yZmTXm4G5mVkIO7mZm\nJeTgbmZWQg7uZmYlpIjodR2QNAHc18ZbHAg8mFN1isT3XS2+72pp5r5fFBF10+r2RXBvl6SxiBjt\ndT26zfddLb7vamn3vt0tY2ZWQg7uZmYlVJbgvrrXFegR33e1+L6rpa37LkWfu5mZPVNZWu5mZpbi\n4G5mVkKFDu6Slkm6S9ImSSt7XZ9OknSRpG2SbkuV7S9praSfJz/362Ud8ybpEEk3SPqZpNslfSQp\nL/t9P0fSjyX9V3Lfn0vKF0j6UfL7fpmkvXpd106QNCBpg6Rrk9dVue97Jd0qaaOksaQs8+96YYO7\npAHgy8CbgSOAkyUd0dtaddTFwLIpZSuB6yNiIXB98rpMdgGnR8QRwCuBDyf/jct+3zuBYyPiZcBi\nYJmkVwJ/DZwXES8BHgFO7WEdO+kj1PZenlSV+wZ4XUQsTs1vz/y7XtjgDhwNbIqIeyLiN8A3geN7\nXKeOiYgbgYenFB8PXJI8vwRY3tVKdVhEbImInybPH6P2P/wI5b/viIhfJS9nJ48AjgWuSMpLd98A\nkg4G/gD4p+S1qMB9TyPz73qRg/sIcH/q9eakrErmRMSW5PkDwJxeVqaTJM0HlgA/ogL3nXRNbAS2\nAWuBu4HtEbErOaWsv+9/B3wCeDJ5fQDVuG+ofYF/T9J6SSuSssy/64XeicmeFhEhqZTzWiXtA1wJ\nfDQiHq015mrKet8RsRtYLGkIuBo4vMdV6jhJbwW2RcR6Scf0uj498OqIGJf0AmCtpDvTB1v9XS9y\ny30cOCT1+uCkrEq2SpoLkPzc1uP65E7SbGqB/esRcVVSXPr7nhQR24EbgFcBQ5ImG2Rl/H1fCrxN\n0r3UulmPBb5E+e8bgIgYT35uo/aFfjRt/K4XObj/BFiYjKTvBZwEXNPjOnXbNcApyfNTgG/1sC65\nS/pbvwrcERFfTB0q+30PJy12JA0Cb6Q23nAD8I7ktNLdd0ScGREHR8R8av8/r4uId1Hy+waQtLek\nfSefA28CbqON3/VCr1CV9BZqfXQDwEURcXaPq9Qxki4FjqGWBnQr8FlgDXA5MI9ayuQTI2LqoGth\nSXo18APgVp7ug/0UtX73Mt/3b1MbPBug1gC7PCI+L+lQai3a/YENwLsjYmfvato5SbfMxyPirVW4\n7+Qer05ezgK+ERFnSzqAjL/rhQ7uZmZWX5G7ZczMrAEHdzOzEnJwNzMrIQd3M7MScnA3MyshB3cz\nsxJycDczK6H/B8KVIz4zdljAAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qwn29SjK-XCg",
        "colab_type": "text"
      },
      "source": [
        "# Split data\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Z0Nl1Oy3eOJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_bbSMUZ3e3m",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-XAIAT-QoKn1",
        "colab_type": "text"
      },
      "source": [
        "Since our task is a regression task, we will randomly split our dataset into **three** sets: train, validation and test data splits.\n",
        "\n",
        "* train: used to train our model.\n",
        "* val : used to validate our model's performance during training.\n",
        "* test: used to do an evaluation of our fully trained model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9wIw3j4Z_0ya",
        "colab_type": "text"
      },
      "source": [
        "<div align=\"left\">\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/lightbulb.gif\" width=\"45px\" align=\"left\" hspace=\"10px\">\n",
        "</div>\n",
        "Splitting the data for classification tasks are a bit different in that we want similar class distributions in each data split. We'll see this in action in the next lesson."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uKmBKodpgHEE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_val_test_split(X, y, val_size, test_size, shuffle):\n",
        "    \"\"\"Split data into train/val/test datasets.\n",
        "    \"\"\"\n",
        "    X_train, X_test, y_train, y_test = train_test_split(\n",
        "        X, y, test_size=test_size, shuffle=shuffle)\n",
        "    X_train, X_val, y_train, y_val = train_test_split(\n",
        "        X_train, y_train, test_size=val_size, shuffle=shuffle)\n",
        "    return X_train, X_val, X_test, y_train, y_val, y_test"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2SFTiscx3gNi",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WuUQwD72NVAE",
        "colab_type": "code",
        "outputId": "2eb09cdd-d96a-4242-80d3-20222efa1e41",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "# Create data splits\n",
        "X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n",
        "    X, y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)\n",
        "print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n",
        "print (f\"X_val: {X_val.shape}, y_test: {y_val.shape}\")\n",
        "print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")\n",
        "print (f\"X_train[0]: {X_train[0]}\")\n",
        "print (f\"y_train[0]: {y_train[0]}\")"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_train: (35, 1), y_train: (35, 1)\n",
            "X_val: (7, 1), y_test: (7, 1)\n",
            "X_test: (8, 1), y_test: (8, 1)\n",
            "X_train[0]: [12.]\n",
            "y_train[0]: [52.50388806]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-Eksj89d_yv",
        "colab_type": "text"
      },
      "source": [
        "# Standardize data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MJVs6JF7trja",
        "colab_type": "text"
      },
      "source": [
        "We need to standardize our data (zero mean and unit variance) in order to optimize quickly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yD7GFSnvsFfg",
        "colab_type": "text"
      },
      "source": [
        "$z = \\frac{x_i - \\mu}{\\sigma}$\n",
        "* $z$ = standardized value\n",
        "* $x_i$ = inputs\n",
        "* $\\mu$ = mean\n",
        "* $\\sigma$ = standard deviation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VlOYPD5GRjRC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.preprocessing import StandardScaler"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CiE3oLCkOCEa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Standardize the data (mean=0, std=1) using training data\n",
        "X_scaler = StandardScaler().fit(X_train)\n",
        "y_scaler = StandardScaler().fit(y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E7dKUyWGJ4Av",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Apply scaler on training and test data\n",
        "standardized_X_train = X_scaler.transform(X_train)\n",
        "standardized_y_train = y_scaler.transform(y_train).ravel()\n",
        "standardized_X_val = X_scaler.transform(X_val)\n",
        "standardized_y_val = y_scaler.transform(y_val).ravel()\n",
        "standardized_X_test = X_scaler.transform(X_test)\n",
        "standardized_y_test = y_scaler.transform(y_test).ravel()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3JC-YFYFJ39Z",
        "colab_type": "code",
        "outputId": "3cfb0813-841b-4640-bbbf-4715e5afab20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "# Check (means should be ~0 and std should be ~1)\n",
        "print (f\"standardized_X_train: mean: {np.mean(standardized_X_train, axis=0)[0]}, std: {np.std(standardized_X_train, axis=0)[0]}\")\n",
        "print (f\"standardized_y_train: mean: {np.mean(standardized_y_train, axis=0)}, std: {np.std(standardized_y_train, axis=0)}\")\n",
        "print (f\"standardized_X_val: mean: {np.mean(standardized_X_val, axis=0)[0]}, std: {np.std(standardized_X_val, axis=0)[0]}\")\n",
        "print (f\"standardized_y_val: mean: {np.mean(standardized_y_val, axis=0)}, std: {np.std(standardized_y_val, axis=0)}\")\n",
        "print (f\"standardized_X_test: mean: {np.mean(standardized_X_test, axis=0)[0]}, std: {np.std(standardized_X_test, axis=0)[0]}\")\n",
        "print (f\"standardized_y_test: mean: {np.mean(standardized_y_test, axis=0)}, std: {np.std(standardized_y_test, axis=0)}\")"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "standardized_X_train: mean: -1.2688263138573217e-16, std: 1.0\n",
            "standardized_y_train: mean: 8.961085841617335e-17, std: 1.0\n",
            "standardized_X_val: mean: 0.455275148918323, std: 0.8941749819083381\n",
            "standardized_y_val: mean: 0.48468545395300805, std: 0.9785039908565315\n",
            "standardized_X_test: mean: -0.44282621906508784, std: 1.0652341142978086\n",
            "standardized_y_test: mean: -0.4940327711822252, std: 1.0650188983159736\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sdruDHf_laWg",
        "colab_type": "text"
      },
      "source": [
        "# From scratch\n",
        "\n",
        "Before we use TensorFlow 2.0 + Keras we will implement linear regression from scratch using NumPy so we can:\n",
        "1. Absorb the fundamental concepts by implementing from scratch\n",
        "2. Appreciate the level of abstraction TensorFlow provides\n",
        "\n",
        "<div align=\"left\">\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/lightbulb.gif\" width=\"45px\" align=\"left\" hspace=\"10px\">\n",
        "</div>\n",
        "\n",
        "It's normal to find the math and code in this section slightly complex. You can still read each of the steps to build intuition for when we implement this using TensorFlow + Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U4Prl8dDl-T4",
        "colab_type": "code",
        "outputId": "55ef92c1-4e73-4a63-ff51-9d3965cf5bc9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "standardized_y_train = standardized_y_train.reshape(-1, 1)\n",
        "print (f\"X: {standardized_X_train.shape}\")\n",
        "print (f\"y: {standardized_y_train.shape}\")"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X: (35, 1)\n",
            "y: (35, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "R8vzjUSW-05x"
      },
      "source": [
        "Our goal is to learn a linear model $\\hat{y}$ that models $y$ given $X$. \n",
        "\n",
        "$\\hat{y} = XW + b$\n",
        "* $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$ ($N$ is the number of samples)\n",
        "* $X$ = inputs | $\\in \\mathbb{R}^{NXD}$ ($D$ is the number of features)\n",
        "* $W$ = weights | $\\in \\mathbb{R}^{DX1}$ \n",
        "* $b$ = bias | $\\in \\mathbb{R}^{1}$ "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R1QzuBFM8by6",
        "colab_type": "text"
      },
      "source": [
        "1.   Randomly initialize the model's weights $W$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KJrD1GiCl-bX",
        "colab_type": "code",
        "outputId": "43c0157f-47a3-4099-d400-50062b6e45b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Initialize random weights\n",
        "W = 0.01 * np.random.randn(INPUT_DIM, 1)\n",
        "b = np.zeros((1, 1))\n",
        "print (f\"W: {W.shape}\")\n",
        "print (f\"b: {b.shape}\")"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W: (1, 1)\n",
            "b: (1, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0IptdjlS8sBw",
        "colab_type": "text"
      },
      "source": [
        "2. Feed inputs $X$ into the model to receive the predictions $\\hat{y}$.\n",
        "  * $\\hat{y} = XW + b$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_RQsyPW4sSLb",
        "colab_type": "code",
        "outputId": "449589ec-558f-4925-8621-ce9f652a97c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Forward pass [NX1] · [1X1] = [NX1]\n",
        "y_hat = np.dot(standardized_X_train, W) + b\n",
        "print (f\"y_hat: {y_hat.shape}\")"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "y_hat: (35, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W5ZTGKol84VO",
        "colab_type": "text"
      },
      "source": [
        "3. Compare the predictions $\\hat{y}$ with the actual target values $y$ using the objective (cost) function to determine the loss $J$. A common objective function for linear regression is mean squarred error (MSE). This function calculates the difference between the predicted and target values and squares it.\n",
        "\n",
        "  * $J(\\theta) = MSE = \\frac{1}{N} \\sum_{i-1}^{N} (y_i - \\hat{y}_i)^2 $\n",
        "    * ${y}$ = ground truth | $\\in \\mathbb{R}^{NX1}$\n",
        "    * $\\hat{y}$ = predictions | $\\in \\mathbb{R}^{NX1}$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cqs_50xcsSkm",
        "colab_type": "code",
        "outputId": "eb89f7c8-63c5-49b3-e94e-654519728168",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Loss\n",
        "N = len(standardized_y_train)\n",
        "loss = (1/N) * np.sum((standardized_y_train - y_hat)**2)\n",
        "print (f\"loss: {loss:.2f}\")"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loss: 1.02\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBQ59qNs90-u",
        "colab_type": "text"
      },
      "source": [
        "4. Calculate the gradient of loss $J(\\theta)$ w.r.t to the model weights.\n",
        "  * $J(\\theta) = \\frac{1}{N} \\sum_i (y_i - \\hat{y}_i)^2  = \\frac{1}{N}\\sum_i (y_i - X_iW)^2 $\n",
        "    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW) X_i = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i) X_i$\n",
        "    * $\\frac{\\partial{J}}{\\partial{W}} = -\\frac{2}{N} \\sum_i (y_i - X_iW)1 = -\\frac{2}{N} \\sum_i (y_i - \\hat{y}_i)1$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NlN9F8bysSiP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Backpropagation\n",
        "dW = -(2/N) * np.sum((standardized_y_train - y_hat) * standardized_X_train)\n",
        "db = -(2/N) * np.sum((standardized_y_train - y_hat) * 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vA8K64TF_fRD",
        "colab_type": "text"
      },
      "source": [
        "5. Update the weights $W$ using a small learning rate $\\alpha$. The simplified intuition is that the gradient tells you the direction for how to increase something so subtracting it will help you go the other way since we want to decrease loss $J(\\theta)$:\n",
        "  * $W = W - \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "  * $b = b - \\alpha\\frac{\\partial{J}}{\\partial{b}}$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nBa96kSXvDnJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Update weights\n",
        "W += -LEARNING_RATE * dW\n",
        "b += -LEARNING_RATE * db"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3b0P4Ls6_thq",
        "colab_type": "text"
      },
      "source": [
        "6. Repeat steps 2 - 5 to minimize the loss and train the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xF6_RVuul-ZJ",
        "colab_type": "code",
        "outputId": "d6dc9f2a-e5ad-465a-d8ed-dd8098bb836a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Initialize random weights\n",
        "W = 0.01 * np.random.randn(INPUT_DIM, 1)\n",
        "b = np.zeros((1, ))\n",
        "\n",
        "# Training loop\n",
        "for epoch_num in range(NUM_EPOCHS):\n",
        "\n",
        "    # Forward pass [NX1] · [1X1] = [NX1]\n",
        "    y_hat = np.dot(standardized_X_train, W) + b\n",
        "\n",
        "    # Loss\n",
        "    loss = (1/len(standardized_y_train)) * np.sum((standardized_y_train - y_hat)**2)\n",
        "\n",
        "    # show progress\n",
        "    if epoch_num%10 == 0:\n",
        "        print (f\"Epoch: {epoch_num}, loss: {loss:.3f}\")\n",
        "\n",
        "    # Backpropagation\n",
        "    dW = -(2/N) * np.sum((standardized_y_train - y_hat) * standardized_X_train)\n",
        "    db = -(2/N) * np.sum((standardized_y_train - y_hat) * 1)\n",
        "\n",
        "    # Update weights\n",
        "    W += -LEARNING_RATE * dW\n",
        "    b += -LEARNING_RATE * db"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 0, loss: 1.025\n",
            "Epoch: 10, loss: 0.040\n",
            "Epoch: 20, loss: 0.028\n",
            "Epoch: 30, loss: 0.028\n",
            "Epoch: 40, loss: 0.028\n",
            "Epoch: 50, loss: 0.028\n",
            "Epoch: 60, loss: 0.028\n",
            "Epoch: 70, loss: 0.028\n",
            "Epoch: 80, loss: 0.028\n",
            "Epoch: 90, loss: 0.028\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fcIUx6d-69_g",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Predictions \n",
        "pred_train = W*standardized_X_train + b\n",
        "pred_test = W*standardized_X_test + b"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WB_c9ek16-FC",
        "colab_type": "code",
        "outputId": "2b9b8568-6e6d-48ec-98d9-cb48fedda2dc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((standardized_y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((standardized_y_test - pred_test) ** 2)\n",
        "print (f\"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}\")"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 0.03, test_MSE: 2.24\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "by9uqJbC699J",
        "colab_type": "code",
        "outputId": "52652814-cfa3-4d39-8d58-a085c2cab0ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Figure size\n",
        "plt.figure(figsize=(15,5))\n",
        "\n",
        "# Plot train data\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plt.scatter(standardized_X_train, standardized_y_train, label='y_train')\n",
        "plt.plot(standardized_X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Plot test data\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plt.scatter(standardized_X_test, standardized_y_test, label='y_test')\n",
        "plt.plot(standardized_X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Show plots\n",
        "plt.show()"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3YAAAE/CAYAAAAKZVEAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXyU9bn///dFjBDXoOBCEKGKtFoo\nYEQrat2j1iOBbxe3qrUU6+/Y09YejlB7tIsKlO5qrdS26HE9tZBaxVKVWtS6gAYFRCoHoTK4sAVF\nA4bk+v1xT5iZZLJMZrlneT0fjzy4P/fcmfu62a5c89nM3QUAAAAAKFy9wg4AAAAAAJAeCjsAAAAA\nKHAUdgAAAABQ4CjsAAAAAKDAUdgBAAAAQIGjsAMAAACAAkdhB+Q5Myszs21mNijsWAAAAJCfKOyA\nDIsWYa1fLWbWGNe+KNX3c/dmd9/L3f+VjXgBAMi1TOfKuPd9zswuzmSsQKHYLewAgGLj7nu1HpvZ\nGkkT3f3xjq43s93cfWcuYgMAIB+kmisBdI0eOyDHzOwGM3vAzO4zs/clXWxmn45+ythgZm+Z2S/N\nrDx6/W5m5mY2ONq+O/r6o2b2vpk9a2ZDQnwkAAAyKjoN4b/NbLWZbTSze8ysMvranmZ2v5ltjubN\n582sr5n9RNIxku6I9vz9JNynAHKLwg4Ix3hJ90raV9IDknZK+oakfpLGSjpL0hWdfP+Fkv5b0n6S\n/iXph9kMFgCAHPtPSWdKOkHSQElNkn4WfW2iglFnVQry5lWSPnL3b0tapKD3b69oGygZFHZAOJ52\n9z+7e4u7N7r7Ind/3t13uvtqSbMkfaaT73/Q3Re7e5OkeySNzEnUAADkxtckTXH39e6+XdL3JX3R\nzExBkddf0mHRvLnI3T8IM1ggHzDHDgjHm/ENM/u4pJ9IOlrSHgr+bT7fyfe/HXf8oaS9OroQAIBC\nEi3eDpE0z8w87qVekvaX9FtJB0l60Mz2knSXpP929+acBwvkEXrsgHB4m/btkpZJOtzd95F0nSTL\neVQAAITM3V1SRNKp7l4Z99XH3Te6+w53v87dPy7pJEmfl3R+67eHFTcQNgo7ID/sLWmrpA/M7BPq\nfH4dAADF7teSppvZIZJkZgeY2b9Fj083syPNrJek9xTMU2+Jft87kj4WRsBA2CjsgPzwbUmXSnpf\nQe/dA+GGAwBAqH4k6XFJC6IrSP9D0ujoa1WS/qQgZy6TNE+xvPkzSZeY2RYz+1FuQwbCZUFvNwAA\nAACgUNFjBwAAAAAFjsIOAAAAAAochR0AAAAAFDgKOwAAAAAocBR2AAAAAFDgdgs7gM7069fPBw8e\nHHYYAIAse/HFFze6e/+w4ygU5EcAKB3dzZF5XdgNHjxYixcvDjsMAECWmdnasGMoJORHACgd3c2R\nDMUEAAAAgAJHYQcAAAAABY7CDgAAAAAKHIUdAAAAABQ4CjsAALLEzH5nZu+a2bIOXj/ZzLaa2ZLo\n13W5jhEAUBzyelVMAAAK3GxJt0i6q5NrnnL3c3MTDgCgWNFjBwBAlrj7Qkmbw44DAFD8KOwAAAjX\np83sZTN71MyOCjsYAEBhYigmAADheUnSoe6+zczOkVQnaWiyC81skqRJkjRo0KDcRQgA6La6+ohm\nzl+p9Q2NGlBZock1w1Q7qion96bHDgDQM42NUlNT2FEUNHd/z923RY/nSSo3s34dXDvL3avdvbp/\n//45jRMA0LW6+oimzlmqSEOjXFKkoVFT5yxVXX0kJ/ensAMApO6KK6Q99pAeeijsSAqamR1kZhY9\nHqMgL28KNyoAQE/MnL9SjU3NCecam5o1c/7KnNyfoZgAgO57/XXpiCMkSb897RLdsKiPBry+IKdD\nTQqJmd0n6WRJ/cxsnaTrJZVLkrv/WtLnJF1pZjslNUo63909pHABAGlY39CY0vlMo7ADAHTPxRdL\n99wjSTr+W/dp/e57S4oNNZFEcdeGu1/Qxeu3KNgOAQBQ4AZUViiSpIgbUFmRk/szFBMA0LnlyyWz\noKibMUNjpz2xq6hrlcuhJgAA5KPJNcNUUV6WcK6ivEyTa4bl5P702AEAknOXJkyQ6uqC9ubNUt++\nWj/lkaSX52qoCQAA+ah11EpYq2JS2AEA2quvl0aPDo5/+Uvp61/f9VLYQ00AAMhXtaOqQpuWkJGh\nmGb2OzN718yWdfD6yWa21cyWRL+uy8R9AQAZ5i6dcUasqNu6NaGok8IfagIAANrLVI/dbAWTv+/q\n5Jqn3P3cDN0PANANKW2U+vzz0nHHBce/+Y00cWLSy8IeagIAANrLSGHn7gvNbHAm3gsAkBmtG6W2\n7qnT4eqVLS3SiSdK//iH1KuX9N570p57dvreYQ41AQAgry1dKj37rDRpUk5vm8tVMT9tZi+b2aNm\ndlQO7wsAJalbG6XeeKNUVhYUdf/zP1Jzc5dFHQAASMJdOu88acQI6Zbc72STq8VTXpJ0qLtvM7Nz\nJNVJGprsQjObJGmSJA0aNChH4QFA8el0o9SmJmn33WMnGxulPn1yFBkAAEUmftGx226Tvva1nIeQ\nkx47d3/P3bdFj+dJKjezfh1cO8vdq929un///rkIDwDyXl19RGOnL9CQKY9o7PQFqquPdPk9Ha1S\n+YPn7o4VdRdcEHzCSFEHAEDq3KWamlhR9957oRR1Uo567MzsIEnvuLub2RgFBeWmXNwbAApdt+fK\ntTG5ZljC9/Xe+ZFW/mRC7IL335f22it7gQMAUMwWLZLGjAmO77hD+spXQg0nI4Wdmd0n6WRJ/cxs\nnaTrJZVLkrv/WtLnJF1pZjslNUo63909E/cGgGLX2Vy5zgq7+NUrr3zgx7p4yaPBC1deKf3qV1mL\nFwCAouYunXKK9Pe/S2bBB6V5MD89U6tiXtDF67co2A4BAJCiTufKdaH20ArVTj0tdoK5dAAA9Nyz\nz0rHHx8c33mndMkl4cYTJ1eLpwAAemhAZYUiSYq4jubQ7dK7t/TRR8FxdXUwZAQAAKSupUUaO1Z6\n7rkgv27eLO2xR9hRJcjldgcAgB6YXDNMFeVlCecqyss0uWZY8m94661gaEhrUbd9O0UdAAA99fTT\nwdZAzz0n3XtvkFfzrKiT6LEDgLwXP1dufUOjBlRWaHLNsOTz68xix2efLc2bl6MoAQAoMi0twYiX\n+npp332lt9/O6+kMFHYAUABqR1V1ulCK3nhD+tjHYu2mJmk3/osHAKBH/vY36dRTg+M//EH63OfC\njacbyPoAUOjie+m+9CXprrvCiwUAgELW3CwNHy6tWCH17y+9+WYwp64AMMcOAArVq68mFnXNzRR1\nAAD01GOPBaNdVqyQ6uqkd98tmKJOoscOAFJWVx/p3ny3bIov6L7xDennP8/t/QEAKBY7d0rDhkmr\nV0sDBwa/lpeHHVXKKOwAIAV19RFNnbN014bhkYZGTZ2zVJJyU9y9+GIwkbtVS0tikQcAALpv3jzp\ns58Njh95RDrnnHDjSQOFHQCkYOb8lbuKulaNTc2aOX9l9gu7+ALu+9+Xrrsu7bfMi95HAAByrakp\nWHRs3TrpsMOk114r+EXHmGMHAClYn2Sj8M7OZ8SDDyYWde4ZK+qmzlmqSEOjXLHex7r6SNrvDQBA\n3rrtNmn33YOibv58adWqgi/qJHrsACAlAyorFElSxA2orMjODeMLuquukm6+OWNvHWrvIwAAufb+\n+9I++wTH5eVSY2Ow8XiRoMcOAFIwuWaYKsoTk0BFeZkm1wzL7I3uuKN9L10GizoppN5HAADCcMEF\nsaLuu9+VPvqoqIo6iR47AEhJa09WVuelxRd03/uedP31mXvvODnvfQQAINcaGqS+fWPtjz4qyBUv\nu4PCDgBSVDuqKjtDFe+7T7rwwljbPfP3iDO5ZljCCp9SlnofAQAIw7hx0kMPBcc33ih95zvhxpNl\nFHYAkA/ie+l++1vp8suzfsuc9D4CAJBrmzZJ/frF2jt3Ft2wy2Qo7AAgTL/+tXTllbF2lnvp2spa\n7yMAAGE44wzp8ceD45/8RLr66nDjySEKOwAIS3wv3WOPSaefHl4sAAAUsnfflQ48MNZubpZ6ldY6\nkaX1tACQD2bMaL/iJUUdAAA9M3ZsrKi79dYgr5ZYUSfRYwcAudM20Tz7rHTcceHFg6wzs99JOlfS\nu+7+ySSvm6RfSDpH0oeSLnP3l3IbJQAUqPXrpaq46QQtLYkfnJaY0itlASAM11yTWNS5U9SVhtmS\nzurk9bMlDY1+TZJ0Ww5iAoDCN3JkrKi7444gr5ZwUSfRYwcA2dW2l+7ll6URI8KLBznl7gvNbHAn\nl4yTdJe7u6TnzKzSzA5297dyEiAAFJp//Us69NBYO8Veurr6SNGuBk2PHQBky1e/2r6XjqIOiaok\nvRnXXhc9BwBo6/DDY0Xd3Xen3EtXVx/R1DlLFWlolEuKNDRq6pylqquPZCfeHKPHDgAybedOqbw8\n1n799SAZAWkws0kKhmtq0KBBIUcDADm0erV02GGxdg/n0s2cv1KNTc0J5xqbmjVz/sqi6LWjxw4A\nMumggxKLOneKOnQmIumQuPbA6Ll23H2Wu1e7e3X//v1zEhwAhO7gg2NF3R/+kNZcuvUNjSmdLzQU\ndgCQCdu3B4nmnXeC9uuv53yzcRSkhyRdYoHjJG1lfh0ASFq5Msirb78dtN2lz30urbccUFmR0vlC\nk5HCzsx+Z2bvmtmyDl43M/ulma0ys1fMbHQm7gsAecFMqohLCvTSIcrM7pP0rKRhZrbOzL5iZl8z\ns69FL5knabWkVZJ+I+n/CylUAMgfe+8tffzjwfGf/pSxD0on1wxTRXlZwrmK8jJNrhmWkfcPW6bm\n2M2WdIukuzp4PX4552MVLOd8bIbuDQDheP99aZ99Yu1IRBowILx4kHfc/YIuXndJ/56jcAAgvy1b\nJg0fHmtneORL6zy6Yl0VMyOFHcs5Ayg5bcf3ZyD5FPMSzAAAdCo+r86fL515ZlZuUzuqqmhza67m\n2LGcM4DisHFjYvLZtCljRV0xL8EMAEBSS5Yk5lX3rBV1xS7vFk8xs0lmttjMFm/YsCHscAAgxkyK\nX43QXdpvv4y8dWdLMAMAUJTMpFGjguMFC1h0LE25KuxYzhlA4Vq3LvHTxPffz3jyKfYlmAEA2OWF\nF9r30p1ySnjxFIlcFXYs5wygoNTVRzR2+oIg8RwS97mUu7TXXhm/X7EvwQwAgKQgrx4bXUPx6afp\npcugTG13wHLOAIpGXX1Ed/6qTs9MPW3XuRFT/qS6l9Zl7Z7FvgQzAKDEPf10+166sWPDi6cIZWpV\nTJZzBlA0akcPVG1ce/A1D0sezIPL1kpaxb4EMwCghMUXdIsWSdXV4cVSxDK1jx0AhCZj2wQ89ZR0\n0km7modN/pOae8V60bI9362Yl2AGAJSgBQuk02KjXxh2mV0UdgAKWus2Aa0rSrZuEyAptSIp7tPE\nFjN97L/+3O4S5rsBANBN8b109fXSyJHhxVIi8m67AwBIRdrbBDz0UGLyaWnRQy++yXw3AAB64i9/\nieXVXr2CXjqKupygxw5AQUtrm4D4gu7QQ6U1aySlP98tY0NDAQAoJPF5ddky6aijwoulBFHYASho\nAyorFElSxHU6bHL2bOnLX461k4z57+l8t4wNDQUAoFA89JA0blxwvO++UkNDuPGUKIZiAigIrfvK\nDZnyiMZOX6C6+oikHmwTYBYr6k44IeMTudMeGgoAQCExixV1K1dS1IWIwg5A3mvtBYs0NMoV6wWr\nq4+odlSVpk0YrqrKCpmkqsoKTZswvH3v2I9/3H7/nKeeynisaQ0NBQCgUPzhD7G8WlUV5NUjjgg3\nphLHUEwAea+zXrDWIZOdDnOML+i+8AXpgQeyFGkPh4YCAFAo3INFUVqtXi0NGRJePNiFHjsAea/H\nvWAXXdS+ly6LRZ3Ug6GhAAAUirvvjhV1RxwR5FWKurxBjx2AvNejXrD4gm78eGnOnCxE1l66K2oC\nAJB32vbS/etf0iGHhBcPkqKwA5D3JtcMS1hpUuqkF+wzn5EWLoy1M7w4Snf0dEVNAADyzh13SF/9\nanA8erT04ovhxoMOUdgByJme7u/W7V6w+F66q66Sbr45k+EDAFA62vbSrV8vHXxwePGgSxR2AHIi\n3f3dOu0FGzxYWrs21g6hlw4AgKJxyy3S178eHJ9wQlZWkUbmsXgKgJzI2v5uZrGi7oYbKOoAAOip\nlpYgr7YWde++S1FXQCjsAORExvd3M2u/4uW11/bsvQAAKHU//rFUFl3V+cwzg7zav3+4MSElDMUE\nkBMZ29+t7Zj/3/xGmjgxzegAAChRzc3SbnElwaZN0n77hRcPeoweOwA5kZH93cwSizr3Lou6uvqI\nxk5foCFTHtHY6QtUVx9JJWwAAIrXDTfEirra2iCvUtQVLHrsAOREWvu7tf00ce7cIAF1Id0FWwAA\nKEpNTdLuu8faDQ3SvvuGFw8ygsIOQMrS2bYg5YIqfh6dlNLiKJ0t2EJhBwAoSddeK910U3B84YXS\nPfeEGw8yhsIOQEpy1gu2fbtUETf/7u9/l046KaW3yPiCLQAAFKodO6Q+fWLt99+X9torvHiQccyx\nA5CSrG1bEM8ssahzT7mokzpemCXlBVsAAChkV18dK+q+8pUgr1LUFR167ACkJKu9YFu3SpWVsfaS\nJdKnPtXjt5tcMyyhd1HqwYItAAAUqsZGaY89Yu0PPkhso6jQYwcgJVnrBTNLLOrc0yrqpGBo6LQJ\nw1VVWSGTVFVZoWkThjO/DgBQ/K68MlbEff3rQV6lqCtq9NgBSEnGe8Hefls6+OBYe/VqaciQNKOM\n6dGCLUAGmdlZkn4hqUzSHe4+vc3rl0maKal1L45b3P2OnAYJoHh88EHiMMvGxsS5dShaGemxM7Oz\nzGylma0ysylJXr/MzDaY2ZLoF7sJAwUqo71gZolFnXtGizogbGZWJulWSWdLOlLSBWZ2ZJJLH3D3\nkdEvijoAPXPppbGi7pprgrxKUVcy0u6xi0taZ0haJ2mRmT3k7q+2ufQBd78q3fsBCF/avWCrVklD\nh8ba77wjHXBA+oEB+WeMpFXuvlqSzOx+SeMktc2RANBz772XuA/djh2J+9ShJGSix25X0nL3jyS1\nJi0AaM8ssahzp6hDMauS9GZce130XFv/z8xeMbMHzeyQ3IQGoCh84Quxou7664O8SlFXkjIxxy5Z\n0jo2yXX/z8xOkvRPSd9y9zeTXAOgWNXXS6NHx9rvvSftvXd48QD548+S7nP3HWZ2haQ7JZ3a9iIz\nmyRpkiQNGjQotxECyD9btkj77RdrNzVJu7F8RinL1aqYf5Y02N1HSHpMQdJKyswmmdliM1u8YcOG\nHIUHIKvMEos6d4o6lIqIpPgeuIGKLZIiSXL3Te6+I9q8Q9LRyd7I3We5e7W7V/fv3z8rwQIoEJ/9\nbKyomz49yKsUdSUvE4VdxpJW9FoSF1AsFiwIirpW27cHyQcoHYskDTWzIWa2u6TzJT0Uf4GZxa0g\npPMkrchhfAAKycaNQV6dNy9o79wZLJICKDOFHUkLQHtm0mmnxdruUu/e4cUDhMDdd0q6StJ8Bbnv\nf919uZn9wMzOi172H2a23MxelvQfki4LJ1oAee2UU6TWTo+f/zzIq2Vl4caEvJJ2n6277zSz1qRV\nJul3rUlL0mJ3f0hB0jpP0k5Jm0XSAorWS9f/RKN/8J+72nWL1qq2mvlAKF3uPk/SvDbnros7nipp\naq7jAlAg3nlHOuigWLu5WeqVq9lUKCQZGYxL0gIgSTJT3Ew6Db7mYVX86VWprIxNwgEASNWYMdKi\nRcHxr38tXXFFuPEgrzHLEkD6brxR+u53dzUH/9efd82ta2xq1sz5KynsAADorkhEGjgw1m5pSZyz\nDiRBYQcgPW0SzeBrHm53yfqGxi7fpq4+opnzV2p9Q6MGVFZocs0wikEAQOn55Cel5cuD49//Xrrs\nslDDQeGgsAPQM//+79KvfhVru2vs9AVSkiJuQGVFp29VVx/R1DlL1djULEmKNDRq6pylkkRxBwAo\nDWvXSoMHx9r00iFFzLwEkFRdfURjpy/QkCmPaOz0Baqrj9vFxCxW1JWV7drCYHLNMFWUJ67QVVFe\npsk1wzq918z5K3cVda1ah3ACAFD0Bg+OFXX33hvkVYo6pIgeOwDtdNSD9pmLzlHfFa/ELmyzJ11r\n71qqQyo7GqrZnSGcAAAUrFWrpKFDY232ekUaKOwAtJOsB23FDWfHGrvvLu3YkfR7a0dVpTx8ckBl\nhSI9GMIJAEDB6t8/2HBckubMkcaPDzceFDwKOwDtxPeUvfLzL2qfHR/EXszCp4mTa4Yl9BBK3RvC\nCQBAwXntNekTn4i16aVDhjDHDkA7rT1la2acu6uoW9lvkMZOeyIr96sdVaVpE4arqrJCJqmqskLT\nJgxn4RQAQHGpqIgVdQ8/TFGHjKLHDkA7z0w9LaE9+JqHVVFepmlZ7EHryRBOAAAKwtKl0ogRsTYF\nHbKAHjsAieJW4Vp2yCc05JqH6UEDAKCnzGJF3V//SlGHrKHHDkCg7bLK7vqkpDdCCQYAgAL30kvS\n0UfH2hR0yDJ67IBS13avnAsvJPkAAJAOs1hR9/e/k1eRE/TYAXmirj6S8v5vaUvSSwcAAHroueek\nT3861iavIofosQPyQOuG4JGGRrliG4LX1Ueyc8OWlsSi7jvfIfkAAJAOs1hR9+yz5FXkHD12QB5I\ntiF4Y1OzZs5fmfleO3rpAADInIULpc98JtYmryIk9NgBeSB+Q/DunO+RHTsSi7q77iL5AACQDrNY\nUbd4MXkVoaLHDsgDAyorFElSxLVuFJ42eukAAMicxx+Xzjgj1iavIg/QYwfkgck1w1RRXpZwrqK8\nTJPT3RB869bEom7+fJIPAADpMIsVdS+/TF5F3qDHDsgDrfPoMroqZhe9dKGswgkAQKGaN0/67GeD\n4969pe3bw40HaIPCDsgTtaOqOiysUirCIhFp4MBYe/HixA1SFVuFs3XBltZVOFvjAAAAceI/LH31\nVekTnwgvFqADDMUE8lxKWyGYJRZ17u2KOqnzVTgBAEDU3Lmxom7//YO8SlGHPEVhB+S5bhVhr76a\n+GniqlWdjvnPySqcAAAUMjNpwoTg+PXXpY0bw40H6AKFHZDnuizCzKSjjoq94C4ddlin79nRapsZ\nW4UTAIBCdf/9sQ9LDz00yKuHHx5uTEA3UNgBea6jYuvcTa8l9tKtX9/tlbmytgonAACFyj3Iqxdc\nELTfeENasybUkIBUUNgBeS5ZEbZmxrm6+Y7/jJ1wlw4+uNvvWTuqStMmDFdVZYVMUlVlhaZNGM7C\nKQCA0nTnnVKv6I/FRx4Z5NXBg0MNCUhVRlbFNLOzJP1CUpmkO9x9epvXe0u6S9LRkjZJ+qK7r8nE\nvYFiF78Vwuh//EU3/3lm7MWtW1X3f+9r5vQFKW9b0NkqnAAAlAT3WEEnSW++mbgIGVBA0i7szKxM\n0q2SzpC0TtIiM3vI3V+Nu+wrkra4++Fmdr6kGZK+mO69gVJRO6pKtaPbJBp3ti0AAKCnbr9d+trX\nguNjjpFeeCHceIA0ZWIo5hhJq9x9tbt/JOl+SePaXDNO0p3R4wclnWbWdvdkAEn99KeJc+l27Ng1\nl45tCwAASFFLS5BXW4u6t96iqENRyERhVyXpzbj2uui5pNe4+05JWyXtn4F7A8XNTPr2t2Ntd2n3\n3Xc12bYAyH9mdpaZrTSzVWY2Jcnrvc3sgejrz5vZ4NxHCZSIX/xCKovOW//MZ4K8etBB4cYEZEje\nLZ5iZpPMbLGZLd6wYUPY4QDh+Na3EnvpmpuTrnjJtgVAfoubrnC2pCMlXWBmR7a5bNd0BUk/UzBd\nAUAmNTcHefWb3wzaGzZITz4ZakhApmWisItIOiSuPTB6Luk1ZrabpH0VLKLSjrvPcvdqd6/u379/\nBsIDCoyZ9POfx9ptJ3bHYdsCIO8xXQEI24wZ0m7RZSXOPjvIq/36hRsTkAWZWBVzkaShZjZEQQF3\nvqQL21zzkKRLJT0r6XOSFrh3c8MtoFRMnCj99rexdjf+icSvmJnqqpgAciLZdIVjO7rG3XeaWet0\nhY05iRAoVjt3SuXlsfbmzVLfvuHFA2RZ2oVdNAldJWm+gu0Ofufuy83sB5IWu/tDkn4r6X/MbJWk\nzQqKPwCt2n44n8LnHmxbAJQGM5skaZIkDRo0KORogDz3/e9L3/tecPy5z0l/+EOo4QC5kJF97Nx9\nnqR5bc5dF3e8XdLnM3EvoKicfrr0xBOxNh3ZQLFJZbrCus6mK7j7LEmzJKm6upr/LIBkPvpI6t07\n1t66Vdpnn/DiAXIo7xZPAUqGGUUdUPx2TVcws90VjFh5qM01rdMVJKYrAD03ZUqsqPvSl4K8SlGH\nEpKRHjsAKRg8WFq7Ntbm5zegaDFdAciBHTukPn1i7W3bpD33DC8eICT02AG5ZBYr6gYOpKgDSoC7\nz3P3I9z9MHe/MXruumhRJ3ff7u6fd/fD3X2Mu68ON2KggHzjG7Gi7oorgrxKUYcSRY8dkAtpLI4C\nAADa+PDDxALuww+lCvZwRWmjxw7Itvii7qSTKOoAAEjHpEmxou6b3wzyKkUdQI8dSlddfSS7+7/R\nSwcAQOZs2ybtvXesvX174gqYQImjxw4lqa4+oqlzlirS0CiXFGlo1NQ5S1VX33YV8uTfO3b6Ag2Z\n8ojGTl+Q/Hvii7pLL6WoAwAgHRdfHCvqpk4N8ipFHZCAHjuUpJnzV6qxqTnhXGNTs2bOX9lpr11r\nQdj6va0FoRRsFE4vHQAAGbR1q1RZGWt/9JFUXh5ePEAeo8cOJWl9Q2NK51t1WBD+5bXEou6//5ui\nDgCAdIwfHyvqvv/9IK9S1AEdoscOJWlAZYUiSYq4AZWdT75OVvitmXFu4gkKOgAAem7zZmn//WPt\npiZpN35kBbpCjx1K0uSaYaooL0s4V1Fepsk1wzr9vvjCb7fmnYlF3a9/TVEHAEA6zjorVtT96EdB\nXqWoA7qFfykoSa3z6FJdFXNyzTBNnbNUK244O+F83UvrMruiJgAApWT9eqkqLo82N0u96H8AUkFh\nh5JVO6oq5WKs9ohK1cYVdZk7WmgAACAASURBVFMu/r6Ou/orFHUAAPTU3nsHWxlI0s03S1ddFW48\nQIGisAO6K8mKl9PDiQQAgMK3dq00eHCsTS8dkBb+9aCodWvPua5s3JhY1D37LHPpAABIh1msqGtd\n8ZKiDkgLPXYoWl3uOdcd7EsHAEDmrFolDR0aa7e0tM+1AHqEj0ZQtDrbhLxLa9YkJprXXqOoAwAg\nHWaxom7mzCCvUtQBGUOPHYpWTzchp5cOAIAMWrFCOvLIWJteOiAr6LFD0epos/EONyF/+eXERBOJ\nUNQBAJAOs1hRd/PN9NIBWUSPHYpW655z8cMxO9yEnF46AAAy5+WXpZEjY23yKpB19NihaNWOqtK0\nCcNVVVkhk1RVWaFpE4YnLpzy5JOJRd2WLSQfAADSYRYr6n7zG/IqkCP02KGodboJOb10AABkzqJF\n0pgxsTZ5FcgpeuxQeh58MLGoa2wk+QAAkA6zWFF3993kVSAE9NihtNBLBwBA5jzzjHTCCbE2eRUI\nDT12KA233ppY1O3cSfIBACAdZrGi7o9/JK8CIUurx87M9pP0gKTBktZI+oK7b0lyXbOkpdHmv9z9\nvHTuC6SEXjoAADJnwQLptNNibfIqkBfS7bGbIukJdx8q6YloO5lGdx8Z/aKoQ2585zuJRV1LC8kH\nAIB0mMWKuocfJq8CeSTdOXbjJJ0cPb5T0pOSrknzPYH00UsHAEDmPPqodM45sTZ5Fcg76fbYHeju\nb0WP35Z0YAfX9TGzxWb2nJnVpnlPoGNf+lJiUedO8gEAIB1msaLuscfIq0Ce6rLHzswel3RQkpeu\njW+4u5tZR//SD3X3iJl9TNICM1vq7v/Xwf0mSZokSYMGDeoqPCCGXjoAADKnrk4aPz7WJq8Cea3L\nws7dT+/oNTN7x8wOdve3zOxgSe928B6R6K+rzexJSaMkJS3s3H2WpFmSVF1dzf8g6Nrhh0v/F/fX\nicQDAEB64j8sXbhQOvHE8GIB0C3pDsV8SNKl0eNLJf2p7QVm1tfMekeP+0kaK+nVNO8LBMxiRd3B\nB1PUAQCQjgceaD+lgaIOKAjpLp4yXdL/mtlXJK2V9AVJMrNqSV9z94mSPiHpdjNrUVBITnd3Cjuk\nh2GXAABkVnxufe456dhjc3LbuvqIZs5fqfUNjRpQWaHJNcNUO6oqJ/cGiklahZ27b5J0WpLziyVN\njB7/Q9LwdO4DJIhPPIcfLr3+enixAEAH2OsVBWP2bOnLX461c/hhaV19RFPnLFVjU7MkKdLQqKlz\ngn8OFHdAatIdignkjln74SEUdQDyF3u9Iv+ZxYq6l17K+QiYmfNX7irqWjU2NWvm/JU5jQMoBhR2\nKAzxBV1NDUMvARSCcQr2eFX0V7b7Qf647bb2H5aOGpXzMNY3NKZ0HkDH0p1jB2RXm7l0Y6c9EYy9\nDykcAEhBSnu9StqpYB56XU6iQ2lyl3rFfa6/bJl01FGhhTOgskKRJEXcgMqKEKIBChs9dshP7glF\n3S2f/oIGX/PwrrH3dfWREIMDgICZPW5my5J8jYu/zt1dUmd7vVZLulDSz83ssA7uNcnMFpvZ4g0b\nNmT2QVAafvazxKLOPdSiTpIm1wxTRXlZwrmK8jJNrhkWUkRA4aLHDnkhfkWsN2acm/Da4GseTmi3\njr1nUjWAsOVyr1f2eUWPte2lW7lSOuKI8OKJ05rLWRUTSB+FHULXuiLWjh0f6Y2ZsQ+5X/6vH6rW\nko/3Z+w9gALQutfrdHWy16ukD919R9xerz/KaZQobjfeKH33u7F2Hs5Rrx1VRSEHZACFHUI3c/5K\nrbjh7IRzg695WFWVFRogMfYeQKFir1eEp20v3RtvSIMHhxYOgOxjjh3CtX27npka2wrxynFTdg29\nXN/QyNh7AAXL3Te5+2nuPtTdT3f3zdHzi6NFndz9H+4+3N0/Ff31t+FGjaLw3e/Giro+fYIij6IO\nKHr02CE8bVa8bDuXbkBlBWPvAQDoruZmabe4H+3efFMaODC8eADkFIUdcq+hQerbd1fz6dsf0Fff\n3EeK26A0vleOsfcAAHTh6quDVS8laf/9pY0bw40HQM5R2CG32vTSyV0nSJoWtyomvXIAAHTTzp1S\neXms/fbb0oEdbZkIoJhR2CE31q+XquIKtZdflkaM2NWkVw4AgBRdcYU0a1ZwPHhwsEAKgJJFYYfs\nS9JLBwAAeuijj6TevWPtDRukfv3CiwdAXmBVTGTPa68lFnVvvEFRBwBAOi6+OFbUDR8e5FWKOgCi\nxw49UNed+XD00gEAkDnbt0sVcXu4btkiVVaGFw+AvEOPHVJSVx/R1DlLFWlolCvYPHzqnKWqq48E\nFzz3XGJRt2EDRR0AAOmYMCFW1B1/fJBXKeoAtEGPHVIyc/5KNcZtSyBJjU3Nmjl/pWpHt9krh4IO\nAICe+/BDac89Y+333pP23ju8eADkNXrskJL1DY3tzp3yf4v0zNTTYie2bctpUVdXH9HY6Qs0ZMoj\nGjt9Qaz3EACAQlVTEyvqzjgjyKsUdQA6QY8dUjKgskKRuOJuzYxzEy/IcS9d69DQ1l7E1qGhktg+\nAQAQqm7NSW/r/felffaJtT/4QNpjj+wGCqAo0GOHlEyuGaaK8jJ9/pW/JhR1f3phTShDLzsbGgoA\nQFi6nJOezPHHx4q68eODvEpRB6Cb6LErYj36pLALtaOq2s2lq3tpXWi9Y8mGhnZ2HgCAXOh0Tnrb\nnNnQIPXtG3dho9SnTw6iBFBM6LErUj36pLArP/1p4oqXLS2Se6hDHgdUVqR0HgCAXOj2B48jRsSK\nuosuCnrpKOoA9ACFXZHK+BBFM+nb34613dvvVReC1qGh8SrKyzS5ZlhIEQEA0I0PHjduDPLo0mBe\nuHbskO6+O0fRAShGFHZFKmNDFP/zPxMLOPe82sagdlSVpk0YrqrKCpmkqsoKTZswnIVTAACh6vSD\nx499TOrfPzh5xRVBXt199xCiBFBMmGNXpNquXhl/vtva9sjlUUEXr3ZUFYUcACCvtOal+Lnu3z26\nr86On6fe1CTtxo9iADIjrR47M/u8mS03sxYzq+7kurPMbKWZrTKzKencE92T1hDF88/P6146AAAK\nQe2oKj0z5VS9Mf2zeuYnX9TZZ44OXvjWt4K8SlEHIIPS/R9lmaQJkm7v6AIzK5N0q6QzJK2TtMjM\nHnL3V9O8NzqR7JPCbq2KGV/QVVVJ69ZlMUoAAIrcunXSIYfE2s3NUi9mwgDIvLQKO3dfIUnW+SIa\nYyStcvfV0WvvlzROEoVdlqU0RPG446Tnn4+16aEDACA9ffoEi6JI0rXXSjfcEG48AIpaLsYAVEl6\nM669TtKxHV1sZpMkTZKkQYMGZTcyBOIL8xNOkJ56KrxYAAAodG+8ESyQ0qqlJS9WkgZQ3LocC2Bm\nj5vZsiRf47IRkLvPcvdqd6/u37piFLLjmGPaz6WjqAMAoOfMYkXdDTfkzfZAAIpflz127n56mveI\nSIobXK6B0XMIU3ySuegi9s4BACAd//ynNCxugTJ66QDkWC5m7y6SNNTMhpjZ7pLOl/RQDu6LZA4+\nuH0vHUUdAAA9ZxYr6n76U3rpAIQi3e0OxpvZOkmflvSImc2Pnh9gZvMkyd13SrpK0nxJKyT9r7sv\nTy9s9IiZ9PbbwfGPfsQCKQAApGP58sQCrqUl2MoAAEKQ7qqYcyXNTXJ+vaRz4trzJM1L516lrK4+\nkvq2BfEKZKNxAAAKRnxu/dWvpCuvDC8WAFBuVsVEGurqI5o6Z6kam5olSZGGRk2ds1SSui7u3BP3\nypk9W7r00ixFCgBA7qT9oWdP1ddLo0fH2nxYCiBPsENmnps5f+Wuoq5VY1OzZs5f2fk3lpcnFnXu\nFHUAgKLQ+qFnpKFRrtiHnnX1WV6bzSxW1P3+9xR1APIKhV2eW9/QmNJ57dwZJJ6dO4P2k0+SeAAA\nRaXHH3r21PPPt1947LLLsnMvAOghhmLmuQGVFYokKeIGVFa0vziFuXShDWEBACBNKX/omY743Hrf\nfdL552f+HgCQAfTY5bnJNcNUUV6WcK6ivEyTa+L2ytm+PTHx1Nd3WdSFMoQFAEqImX3ezJabWYuZ\nVXdy3VlmttLMVpnZlFzGWKiSfrjZyfkeefTR9r10FHUA8hiFXZ6rHVWlaROGq6qyQiapqrJC0yYM\nj/WumUkVcYnMXRo5stP3zPkQFgAoTcskTZC0sKMLzKxM0q2SzpZ0pKQLzOzI3IRXuLr1oWc6zKRz\noot7z57NlAYABYGhmAWgdlRV+2GS770n7btvrP3669Lhh3fr/XI6hAUASpS7r5Ak63yj6jGSVrn7\n6ui190saJ+nVrAdYwFpzYsanFMydK02YsKtZ99K64B5THmHaAoC8R2FXiNLcly6leXsAgGyqkvRm\nXHudpGNDiqWgJP3QMx3xufX++1V3xAk9324IAELAUMxC8s47iYknEunR8JCsD2EBgBJhZo+b2bIk\nX+OycK9JZrbYzBZv2LAh029fuu69t/1cui9+kWkLAAoOPXaFIs1eunhZG8ICACXG3U9P8y0ikg6J\naw+Mnkt2r1mSZklSdXU1k74yIT63/ulP0nnn7WoybQFAoaGwy3fr10tVcQXXli1SZWXab5vxISwA\ngJ5YJGmomQ1RUNCdL+nCcEMqAb/9rTRxYqyd5MNSpi0AKDQMxcxnZolFnXtGijoAQPaZ2XgzWyfp\n05IeMbP50fMDzGyeJLn7TklXSZovaYWk/3X35WHFXBLMYkXdX/7S4QgYpi0AKDT02OWjVaukoUNj\n7cZGqU+f8OIBAKTM3edKmpvk/HpJ58S150mal8PQStMtt0hf/3qs3cWUBqYtACg0FHb5Jn68f//+\n0rvvhhcLAADFID63Pvmk9JnPdOvbmLYAoJAwFDNfvPxyYuJpaqKoAwAgHT/6UfsVL7tZ1AFAoaHH\nLoPq6iM9G7IRn3Q+9SlpyZLsBQkAQCmIz63PPisdd1x4sQBADtBjlyF19RFNnbNUkYZGuWIbmdbV\nJ121OvD004mJp6WFog4AgHRcf337XjqKOgAlgMIuQ1LeyNRMOvHE4Piss4LE03avOgAA0D2tefQH\nPwjaL72U1p6vAFBoKOwypNsbmT76aPteukcfzWJkAAAUucmTpV5xP9K4S6NGhRcPAISAOXYZ0q2N\nTOMLuksuke68MweRAQBQpNwTC7ply6SjjgovHgAIET12Kaqrj2js9AUaMuURjZ2+YNccuk43Mn3i\nifbj/SnqAADoud/8pn0vHUUdgBJGj10KWhdIaZ1L17pAitTJRqajB8be4Fe/kq68MudxAwBQNFpa\npLK4D1LXr5cOPji8eAAgT1DYpaCzBVJaNzHdtb1BXZ0UX9QxgRsAgPTcfLP0H/8RHJ94orRwYbjx\nAEAeobBLQbcXSIkfdnnPPdKFF2YxKgAAilzbXrp335X69w8vHgDIQ2nNsTOzz5vZcjNrMbPqTq5b\nY2ZLzWyJmS1O555hSlgIJdn5v/+9/Vw6ijoAAHpu5sxYUVdTE+RWijoAaCfdHrtlkiZIur0b157i\n7hvTvF+oJtcMS5hjJ8UtkBJf0C1eLB19dAgRAgBQJHbulMrLY+1Nm6T99gsvHgDIc2n12Ln7Cnfv\nYAfu4lM7qkrTJgxXVWWFTFJVZYV+e+CGxAVS3CnqAABIxw9/GCvqxo8PcitFHQB0Kldz7FzSX83M\nJd3u7rNydN+MS1ggJb6Xjr1zAABIT1OTtPvusXZDg7TvvuHFAwAFpMseOzN73MyWJfkal8J9TnD3\n0ZLOlvTvZnZSJ/ebZGaLzWzxhg0bUrhFDv3xj7Gibr/92DsHAIB0fec7saLuoouC3EpRBwDd1mWP\nnbufnu5N3D0S/fVdM5sraYykpGsUR3vzZklSdXV1fu0R4J64GeqqVdJhh4UXDwAAhW7HDqlPn1j7\n/felvfYKLx4AKFBpzbHrDjPb08z2bj2WdKaCRVcKy513xoq6oUODIo+iDgCAnrvttlhRN3FikFsp\n6gCgR9KaY2dm4yXdLKm/pEfMbIm715jZAEl3uPs5kg6UNNeCoYu7SbrX3f+SZty507aX7s03pYED\nO74eAAB0rqkp+JB07dqg/eGHUkXyLYUAAN2T7qqYc919oLv3dvcD3b0men59tKiTu692909Fv45y\n9xszEXhO3HprrKg79tigyEujqKurj2js9AUaMuURjZ2+QHX1kQwFCgBAgfjzn4O5dGvXSvPmBbmV\nog4A0parVTELS3OztFvcb80770gHHJDWW9bVRxL2wIs0NGrqnKWSFFtlEwCAYvXRR9Khh0pvvy0d\ncYS0fHlirgUApCXrc+wKzrRpsURz1lnBJ4lpFnWSNHP+yoSNzSWpsalZM+eXzDaAAIBSNXeu1Lt3\nUNQ99pi0ciVFHQBkGP+rtmq7d87mzVLfvhl7+/UNjSmdBwCg4O3YIR18sLRli/TJT0pLlkhlZWFH\nBQBFiR47KXHvnAsuCHrpMljUSdKAyuTzBzo6DwBAQXvggWDFyy1bpL/9TVq6lKIOALKoZHvs6uoj\n+sXDS/W3686Onczi3jmTa4YlzLGTpIryMk2uGZaV+wEIX1NTk9atW6ft27eHHUre6NOnjwYOHKjy\n8vKwQ0G2NDZK/foFK10efbT0wguJq0sDQBvky0C6ObIkC7u6+ojmz7hDf3vge5Kku0eerRvP/Q9N\ne32rakdlp7BrXSBl5vyVWt/QqAGVFZpcM4yFU4Aitm7dOu29994aPHiwolu+lDR316ZNm7Ru3ToN\nGTIk7HCQDffcI118cXD81FPSCSeEGw+AgkC+zEyOLL3C7qOPdOJJw1W7bYvW7XOATpl0u5rKyqXo\nQibZLLRqR1VRyAElZPv27SWdpNoyM+2///7asGFD2KEg0z74QNpnH6mlRRo7Vlq4kF46AN1GvsxM\njiyt/3UffFDq3Vv7b9uii7/wQ51w5e+Coi6KhUwAZFopJ6lk+P0oQrNnB9MYWlqkZ5+Vnn6aog5A\nysgP6f8elMb/vI2N0t57S5//vDRypE688a96esiodpexkAkAdGzw4MHauHFj2tegSGzbJplJX/6y\ndOqpQWF33HFhRwUAoQozVxZ/YXfffdIeewQJaOFCqb5e3z77SFWUJ67MxUImAIBMMrPPm9lyM2sx\ns+pOrltjZkvNbImZLc5ljD02a1bwgakkLVokPfFEUOQBAEJT/IXdT38qHX+81NwsnXiipGCu27QJ\nw1VVWSGTVFVZoWkThjP/DUDRWbNmjT7+8Y/rsssu0xFHHKGLLrpIjz/+uMaOHauhQ4fqhRde0ObN\nm1VbW6sRI0bouOOO0yuvvCJJ2rRpk84880wdddRRmjhxotx91/vefffdGjNmjEaOHKkrrrhCzc3N\nHYVQypZJmiBpYTeuPcXdR7p7hwVgXnjvvaCAu+IK6ayzgl666vwOGQC6Uiy5svgLu0WLpGeeaTfe\nv3ZUlZ6ZcqremP5ZPTPlVIo6AEVr1apV+va3v63XXntNr732mu699149/fTT+vGPf6ybbrpJ119/\nvUaNGqVXXnlFN910ky655BJJ0ve//32dcMIJWr58ucaPH69//etfkqQVK1bogQce0DPPPKMlS5ao\nrKxM99xzT5iPmJfcfYW7rww7joy55RZp332D4/p66dFH6aUDUDSKIVeW3qqYABCWbPwQHPfJYEeG\nDBmi4cOHS5KOOuoonXbaaTIzDR8+XGvWrNHatWv1xz/+UZJ06qmnatOmTXrvvfe0cOFCzZkzR5L0\n2c9+Vn379pUkPfHEE3rxxRd1zDHHSJIaGxt1wAEHZP7ZSodL+quZuaTb3X1W2AElaGiQon/2GjdO\nmjs3I3+X6+ojbAEEILkQ8mUx5EoKOwDIlW4UYdnQu3fvXce9evXa1e7Vq5d27tyZ8kao7q5LL71U\n06ZNy2ichcjMHpd0UJKXrnX3P3XzbU5w94iZHSDpMTN7zd3bDd80s0mSJknSoEGDehxzSn72M+nq\nq4PjV16Roj/0pKuuPqKpc5aqsSkYlhRpaNTUOUslieIOQCj5shhyZfEPxQQAdOrEE0/cNTzkySef\nVL9+/bTPPvvopJNO0r333itJevTRR7VlyxZJ0mmnnaYHH3xQ7777riRp8+bNWrt2bTjBh8zdT3f3\nTyb56m5RJ3ePRH99V9JcSWM6uG6Wu1e7e3X//v0z8wAd2bw5+MT86qulL3wh+CErQ0WdJM2cv3JX\nUdeqMbqfLADko0LIlfTYAUCJ+973vqfLL79cI0aM0B577KE777xTknT99dfrggsu0FFHHaXjjz9+\nVy/RkUceqRtuuEFnnnmmWlpaVF5erltvvVWHHnpomI9RkMxsT0m93P396PGZkn4QalAzZkhTpgTH\nr74qfeITGb9FR/vGsp8sgHxVCLnSPKShQd1RXV3tixcXxsrPANDWihUr9Iks/FBc6JL9vpjZi3m/\nImSKzGy8pJsl9ZfUIGmJu9eY2QBJd7j7OWb2MQW9dFLwYeu97n5jV++dlfy4YYPUOv/j4oul//mf\nzL5/nLHTFyiSpIirqqzQM1NOzdp9AeQn8mVMOjmSoZgAAGSBu89194Hu3tvdD3T3muj59e5+TvR4\ntbt/Kvp1VHeKuqz44Q9jRd3KlVkt6iRpcs0w9pMFgAxjKCYAAKXqnXekg6Jrv3zlK9Idd+Tktq0L\npLAqJgBkDoUdAACl6L33YkXdqlXSYYfl9Pa1o6oo5AAggxiKCQBAKdprL+mpp4IVL3Nc1AEAMo/C\nDgCAUtSrl3TCCWFHAQDIEAo7AAAAAChwFHYAAAAA0IU1a9bs2oy8J2666aYMRtMehR0AlLh0EtXx\nxx+f4WgAAMhPRV3YmdlMM3vNzF4xs7lmVtnBdWeZ2UozW2VmU9K5JwAUq7r6iMZOX6AhUx7R2OkL\nVFcfycl9O0tUO3fu7PR7//GPf2QjJAAAOpTpfHndddfp5z//+a72tddeq1/84hftrpsyZYqeeuop\njRw5Uj/72c/U3NysyZMn65hjjtGIESN0++23S5LeeustnXTSSRo5cqQ++clP6qmnntKUKVPU2Nio\nkSNH6qKLLkor3o6ku93BY5KmuvtOM5shaaqka+IvMLMySbdKOkPSOkmLzOwhd381zXsDQNGoq49o\n6pylamxqliRFGho1dc5SSerxkvDXXXed9ttvP33zm9+UFCSqAw44QN/4xjcSrpsyZYpWrFihkSNH\n6tJLL1Xfvn01Z84cbdu2Tc3NzXrkkUc0btw4bdmyRU1NTbrhhhs0btw4SdJee+2lbdu26cknn9T3\nvvc99evXT8uWLdPRRx+tu+++W2bW098SAADayUa+vPzyyzVhwgR985vfVEtLi+6//3698MIL7a6b\nPn26fvzjH+vhhx+WJM2aNUv77ruvFi1apB07dmjs2LE688wzNWfOHNXU1Ojaa69Vc3OzPvzwQ514\n4om65ZZbtGTJkh4+edfSKuzc/a9xzeckfS7JZWMkrXL31ZJkZvdLGicp64VdXX2EzU8BFISZ81fu\nSlKtGpuaNXP+ypwnqtmzZ+ull17SK6+8ov322087d+7U3Llztc8++2jjxo067rjjdN5557Ur2urr\n67V8+XINGDBAY8eO1TPPPKMTWHUxL5EfARSqbOTLwYMHa//991d9fb3eeecdjRo1Svvvv3+X3/fX\nv/5Vr7zyih588EFJ0tatW/X666/rmGOO0eWXX66mpibV1tZq5MiRPYorVZncoPxySQ8kOV8l6c24\n9jpJx2bwvkllo5oHgGxZ39CY0vnu6GmikqQzzjhD++23nyTJ3fWd73xHCxcuVK9evRSJRPTOO+/o\noNbNraPGjBmjgQMHSpJGjhypNWvWUNjlIfIjgEKWjXwpSRMnTtTs2bP19ttv6/LLL+/W97i7br75\nZtXU1LR7beHChXrkkUd02WWX6eqrr9Yll1ySVnzd0eUcOzN73MyWJfkaF3fNtZJ2Sron3YDMbJKZ\nLTazxRs2bOjx+3RWzQNAvhlQWZHS+e5qTVS///3vu52oJGnPPffcdXzPPfdow4YNevHFF7VkyRId\neOCB2r59e7vv6d27967jsrKyLufnIRzkRwCFLFv5cvz48frLX/6iRYsWJS3UJGnvvffW+++/v6td\nU1Oj2267TU1NTZKkf/7zn/rggw+0du1aHXjggfrqV7+qiRMn6qWXXpIklZeX77o2G7rssXP30zt7\n3cwuk3SupNPc3ZNcEpF0SFx7YPRcR/ebJWmWJFVXVyd7v27JVjUPANkwuWZYQi+KJFWUl2lyzbC0\n3nf8+PG67rrr1NTU1OECKW0TVVtbt27VAQccoPLycv3tb3/T2rVr04oJ4SI/Aihk2cqXu+++u045\n5RRVVlaqrKws6TUjRoxQWVmZPvWpT+myyy7TN77xDa1Zs0ajR4+Wu6t///6qq6vTk08+qZkzZ6q8\nvFx77bWX7rrrLknSpEmTNGLECI0ePVr33JN2f1g7aQ3FNLOzJP2XpM+4+4cdXLZI0lAzG6KgoDtf\n0oXp3Lc7BlRWKJIkSaVbzQNANrQOgcv0vKeeJKq+ffsmvH7RRRfp3/7t3zR8+HBVV1fr4x//eFox\nIVzkRwCFLFv5sqWlRc8995z+8Ic/dHhNeXm5FixYkHDupptuareNwaWXXqpLL7203ffPmDFDM2bM\nSCvOzqQ7x+4WSb0lPRadRP+cu3/NzAZIusPdz4mumHmVpPmSyiT9zt2Xp3nfLmWrmgeAbKkdVZXx\nOU49TVSXXXbZruN+/frp2WefTfq927ZtkySdfPLJOvnkk3edv+WWW3oeNLKK/Aig0GU6X7766qs6\n99xzNX78eA0dOjRj75tr6a6KeXgH59dLOieuPU/SvHTulapsVfMAUCiKJVEhs8iPAJDoyCOP1OrV\nq3e1ly5dqi996UsJ1/Tu3VvPP/98rkNLSSZXxcw72fj0GwAKRbEkKmQe+REAOjZ8+PCs7jeXLUVd\n2AEAYgo1UQEAgK51j6cJbwAABqFJREFUud0BAKDnki8WXLr4/QAAJEN+SP/3gMIOALKkT58+2rRp\nE8kqyt21adMm9enTJ+xQAAB5hHyZmRzJUEwAyJKBAwdq3bp12rBhQ9ih5I0+ffpo4MCBYYcBAMgj\n5MtAujmSwg4AsqS8vFxDhgwJOwwAAPIa+TIzGIoJAAAAAAWOwg4AAAAAChyFHQAAAAAUOMvn1WfM\nbIOktWHH0U39JG0MO4gMKJbnkIrnWYrlOaTieRaeI/MOdff+YQdRKELOj/n09yZbSuEZpdJ4zlJ4\nRonnLCbJnrFbOTKvC7tCYmaL3b067DjSVSzPIRXPsxTLc0jF8yw8B0pZKfy9KYVnlErjOUvhGSWe\ns5ik84wMxQQAAACAAkdhBwAAAAAFjsIuc2aFHUCGFMtzSMXzLMXyHFLxPAvPgVJWCn9vSuEZpdJ4\nzlJ4RonnLCY9fkbm2AEAAABAgaPHDgAAAAAKHIVdD5nZ581suZm1mFmHK9eY2RozW2pmS8xscS5j\n7I4UnuMsM1tpZqvMbEouY+wuM9vPzB4zs9ejv/bt4Lrm6J/HEjN7KNdxdqSr32Mz621mD0Rff97M\nBuc+yq514zkuM7MNcX8GE8OIsytm9jsze9fMlnXwupnZL6PP+YqZjc51jN3Rjec42cy2xv15XJfr\nGJHfiiXfdaaYcmFnCj1PdqZYcmhXiiXHdqZY8m9nspWbKex6bpmkCZIWduPaU9x9ZJ4uz9rlc5hZ\nmaRbJZ0t6UhJF5jZkbkJLyVTJD3h7kMlPRFtJ9MY/fMY6e7n5S68jnXz9/grkra4++GSfiZpRm6j\n7FoKf1ceiPszuCOnQXbfbElndfL62ZKGRr8mSbotBzH1xGx1/hyS9FTcn8cPchATCkux5LvOFFMu\n7EzB5snOFEsO7UqR5djOzFZx5N/OzFYWcjOFXQ+5+wp3Xxl2HOnq5nOMkbTK3Ve7+0eS7pc0LvvR\npWycpDujx3dKqg0xllR15/c4/vkelHSamVkOY+yOQvm70iV3XyhpcyeXjJN0lweek1RpZgfnJrru\n68ZzAJ0qlnzXmSLLhZ0p5DzZmWLJoV0phr+DXSqW/NuZbOVmCrvsc0l/NbMXzWxS2MH0UJWkN+Pa\n66Ln8s2B7v5W9PhtSf9/e/fvItUVhnH8+4AYQYJERY0YQhYCgpUg4o9UEiwshJAUqWJhoUX+A7s0\ngn9AKnsLBX+BIoqxlCQIy6IRk02VRVaSwmAjCm+Ke1YG45y5zs7eO+fM84HL3vmxw/veO5dnz50z\nd7cPed4GSb9Kui9pWkKtzTZ+85yIeA08B7Z0Ul17bd8rX6fpE5ckfdJNaRNXynHRxkFJ85JuStrT\ndzFWrBryLqeGY77knMypJUNHmaWMzanhWGzjvbN53VpXVDJJd4Ad73joTERcbfkyX0TEkqRtwG1J\nj9MovTMT6mMq5HoZvBERIWnYJV8/TftkDrgraSEiFiddqw11HbgQES8lnaI5g3qk55pm2QOaY+KF\npGPAFZrpLTZDasm7nJqyMMc5OfOcsXUYK5s9sMuIiC8n8BpL6eczSZdpPkbvNOgm0McSMHjGZ1e6\nr3O5XiQtS/o4Ip6mj+SfDXmNlX3yp6R7wF6g78Bqs41XnvOXpHXAJuCfbsprbWQfETFY83ngXAd1\nrYWpOS5WIyL+HVi/IelHSVsj4u8+67Ju1ZJ3OTVlYU7FOZlTS4aOMksZm1PEsbga42azp2KuIUkb\nJX24sg4cpfmCdml+AT6X9Jmk9cC3wDReJesacCKtnwD+dwZW0keSPkjrW4HDwKPOKhyuzTYe7O8b\n4G5M3z+iHNnHW/PgjwO/dVjfJF0DvktX5zoAPB+Y4lQMSTtWvmciaT9NLpT2x471rKK8yyklC3NK\nzsmcWjJ0lFnK2Jwq8jdn7GyOCC9jLMBXNHN6XwLLwK10/07gRlqfA+bT8pBmukfvtb9vH+n2MeAJ\nzRm7qesj1biF5ipfvwN3gM3p/n3A+bR+CFhI+2QBONl33bltDPwAHE/rG4CLwB/Az8Bc3zWP2cfZ\ndDzMAz8Bu/uueUgfF4CnwKt0jJwETgOn0+OiuTrZYnov7eu75jH7+H5gf9wHDvVds5fpWmrJu9X2\nmG5PfRaO6LPonBzRWxUZOoE+i8jYET1Wkb+r7HGsbFb6ZTMzMzMzMyuUp2KamZmZmZkVzgM7MzMz\nMzOzwnlgZ2ZmZmZmVjgP7MzMzMzMzArngZ2ZmZmZmVnhPLAzMzMzMzMrnAd2ZmZmZmZmhfPAzszM\nzMzMrHD/AS/1Tgqnoy8LAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 1080x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S3MlFkZ8_der",
        "colab_type": "text"
      },
      "source": [
        "Since we standardized our inputs and outputs, our weights were fit to those standardized values. So we need to unstandardize our weights so we can compare it to our true weight (3.5).\n",
        "\n",
        "Note that both X and y were standardized.\n",
        "\n",
        "$\\hat{y}_{scaled} = b_{scaled} + \\sum_{j=1}^{k}W_{{scaled}_j}x_{{scaled}_j}$\n",
        "* $y_{scaled} = \\frac{\\hat{y} - \\bar{y}}{\\sigma_y}$\n",
        "* $x_{scaled} = \\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n",
        "\n",
        "$\\frac{\\hat{y} - \\bar{y}}{\\sigma_y} = b_{scaled} + \\sum_{j=1}^{k}W_{{scaled}_j}\\frac{x_j - \\bar{x}_j}{\\sigma_j}$\n",
        "\n",
        "$ \\hat{y}_{scaled} = \\frac{\\hat{y}_{unscaled} - \\bar{y}}{\\sigma_y} = {b_{scaled}} + \\sum_{j=1}^{k} {W}_{{scaled}_j} (\\frac{x_j - \\bar{x}_j}{\\sigma_j}) $\n",
        "\n",
        "$\\hat{y}_{unscaled} = b_{scaled}\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} {W}_{{scaled}_j}(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j + \\sum_{j=1}^{k}{W}_{{scaled}_j}(\\frac{\\sigma_y}{\\sigma_j})x_j $\n",
        "\n",
        "In the expression above, we can see the expression $\\hat{y}_{unscaled} = W_{unscaled}x + b_{unscaled} $ where\n",
        "\n",
        "* $W_{unscaled} = \\sum_{j=1}^{k}{W}_j(\\frac{\\sigma_y}{\\sigma_j}) $\n",
        "\n",
        "* $b_{unscaled} = b_{scaled}\\sigma_y + \\bar{y} - \\sum_{j=1}^{k} {W}_j(\\frac{\\sigma_y}{\\sigma_j})\\bar{x}_j$\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C-PcjG4G2TRq",
        "colab_type": "code",
        "outputId": "74ac88ec-1c1a-4747-bd82-7902f9318bf7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        }
      },
      "source": [
        "# Unscaled weights\n",
        "W_unscaled = W * (y_scaler.scale_/X_scaler.scale_)\n",
        "b_unscaled = b * y_scaler.scale_ + y_scaler.mean_ - np.sum(W_unscaled*X_scaler.mean_)\n",
        "print (\"[actual] y = 3.5X + noise\")\n",
        "print (f\"[model] y_hat = {W_unscaled[0][0]:.1f}X + {b_unscaled[0]:.1f}\") "
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[actual] y = 3.5X + noise\n",
            "[model] y_hat = 3.4X + 9.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LD5m4v1A9nqF",
        "colab_type": "text"
      },
      "source": [
        "Now let's implement linear regression with TensorFlow + Keras."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "33NIijOMKqZF",
        "colab_type": "text"
      },
      "source": [
        "# TensorFlow + Keras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wCuIxAH7KuwG",
        "colab_type": "text"
      },
      "source": [
        "We will be using [Dense layers](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers/Dense) in our MLP implementation. The layer applies an activation function on the dot product of the layer's inputs and its weights.\n",
        "\n",
        "$ z = \\text{activation}(XW)$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QvjSlcsfKqjY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Input"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "07hfw6dcK_jo",
        "colab_type": "code",
        "outputId": "c05d40b6-a1ec-4bfc-b72f-c102fde37ecd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "x = Input(shape=(INPUT_DIM,))\n",
        "fc = Dense(units=HIDDEN_DIM, activation='linear')\n",
        "z = fc(x)\n",
        "W, b = fc.weights\n",
        "print (f\"z {z.shape} = x {x.shape} · W {W.shape} + b {b.shape}\")"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "z (None, 1) = x (None, 1) · W (1, 1) + b (1,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4drnbzryeVsD",
        "colab_type": "text"
      },
      "source": [
        "# Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sGH_pQaDOb49",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Input\n",
        "from tensorflow.keras.losses import MeanSquaredError\n",
        "from tensorflow.keras.metrics import MeanAbsolutePercentageError\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.optimizers import Adam"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V-3zRdoIeaas",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Linear regression\n",
        "class LinearRegression(Model):\n",
        "    def __init__(self, hidden_dim):\n",
        "        super(LinearRegression, self).__init__()\n",
        "        self.fc1 = Dense(units=hidden_dim, activation='linear')\n",
        "        \n",
        "    def call(self, x_in, training=False):\n",
        "        \"\"\"Forward pass.\"\"\"\n",
        "        y_pred = self.fc1(x_in)\n",
        "        return y_pred\n",
        "    \n",
        "    def sample(self, input_shape):\n",
        "        x_in = Input(shape=input_shape)\n",
        "        return Model(inputs=x_in, outputs=self.call(x_in)).summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8y3YIttkeaX2",
        "colab_type": "code",
        "outputId": "c1b26629-a5b8-47bc-c446-4a85de7e2c0b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "# Initialize the model\n",
        "model = LinearRegression(hidden_dim=HIDDEN_DIM)\n",
        "model.sample(input_shape=(INPUT_DIM,))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 1)                 2         \n",
            "=================================================================\n",
            "Total params: 2\n",
            "Trainable params: 2\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yj5snqjN0Y4j",
        "colab_type": "text"
      },
      "source": [
        "When we implemented linear regression from scratch, we used batch gradient descent to update our weights. But there are actually many different [gradient descent optimization algorithms](https://ruder.io/optimizing-gradient-descent/) to choose from and it depends on the situation. However, the [ADAM optimizer](https://ruder.io/optimizing-gradient-descent/index.html#gradientdescentoptimizationalgorithms/#adam) has become a standard algorithm for most cases. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0eZiOOYeerCV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compile\n",
        "model.compile(optimizer=Adam(lr=LEARNING_RATE),\n",
        "              loss=MeanSquaredError(),\n",
        "              metrics=[MeanAbsolutePercentageError()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ysKlKW_lki9R",
        "colab_type": "text"
      },
      "source": [
        "<img width=\"45\" src=\"http://bestanimations.com/HomeOffice/Lights/Bulbs/animated-light-bulb-gif-29.gif\" align=\"left\" vspace=\"5px\" hspace=\"10px\">\n",
        "\n",
        "Here are the full list of options for [optimizer](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers), [loss](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/losses) and [metrics](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/metrics)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VtxXsNq3hrz",
        "colab_type": "text"
      },
      "source": [
        "# Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PWn-wFPEFu38",
        "colab_type": "text"
      },
      "source": [
        "When we implemented linear regression from scratch, we used batch gradient descent to update our weights. This means that we calculated the gradients using the entire training dataset. We also could've updated our weights using stochastic gradient descent (SGD) where we pass in one training example at a time. The current standard is mini-batch gradient descent, which strikes a balance between batch and stochastic GD, where we update the weights using a mini-batch of n (`BATCH_SIZE`) samples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u_ZOiIGleq_l",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "81046735-603d-48ce-abc3-0c4b42b88ade"
      },
      "source": [
        "# Training\n",
        "model.fit(x=standardized_X_train, \n",
        "          y=standardized_y_train,\n",
        "          validation_data=(standardized_X_val, standardized_y_val),\n",
        "          epochs=NUM_EPOCHS,\n",
        "          batch_size=BATCH_SIZE,\n",
        "          shuffle=False,\n",
        "          verbose=1)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 35 samples, validate on 7 samples\n",
            "Epoch 1/100\n",
            "35/35 [==============================] - 1s 29ms/sample - loss: 2.2303 - mean_absolute_percentage_error: 189.2383 - val_loss: 1.8631 - val_mean_absolute_percentage_error: 110.1029\n",
            "Epoch 2/100\n",
            "35/35 [==============================] - 0s 604us/sample - loss: 1.1683 - mean_absolute_percentage_error: 129.5676 - val_loss: 0.8827 - val_mean_absolute_percentage_error: 85.6471\n",
            "Epoch 3/100\n",
            "35/35 [==============================] - 0s 650us/sample - loss: 0.5139 - mean_absolute_percentage_error: 72.2302 - val_loss: 0.3088 - val_mean_absolute_percentage_error: 63.4733\n",
            "Epoch 4/100\n",
            "35/35 [==============================] - 0s 675us/sample - loss: 0.1775 - mean_absolute_percentage_error: 58.3600 - val_loss: 0.0744 - val_mean_absolute_percentage_error: 43.0542\n",
            "Epoch 5/100\n",
            "35/35 [==============================] - 0s 635us/sample - loss: 0.0590 - mean_absolute_percentage_error: 55.3533 - val_loss: 0.0241 - val_mean_absolute_percentage_error: 29.1076\n",
            "Epoch 6/100\n",
            "35/35 [==============================] - 0s 724us/sample - loss: 0.0501 - mean_absolute_percentage_error: 52.8501 - val_loss: 0.0298 - val_mean_absolute_percentage_error: 21.7504\n",
            "Epoch 7/100\n",
            "35/35 [==============================] - 0s 718us/sample - loss: 0.0740 - mean_absolute_percentage_error: 54.2517 - val_loss: 0.0345 - val_mean_absolute_percentage_error: 17.8960\n",
            "Epoch 8/100\n",
            "35/35 [==============================] - 0s 685us/sample - loss: 0.0897 - mean_absolute_percentage_error: 51.1155 - val_loss: 0.0292 - val_mean_absolute_percentage_error: 17.8972\n",
            "Epoch 9/100\n",
            "35/35 [==============================] - 0s 693us/sample - loss: 0.0826 - mean_absolute_percentage_error: 46.4416 - val_loss: 0.0204 - val_mean_absolute_percentage_error: 16.6245\n",
            "Epoch 10/100\n",
            "35/35 [==============================] - 0s 745us/sample - loss: 0.0588 - mean_absolute_percentage_error: 41.9437 - val_loss: 0.0161 - val_mean_absolute_percentage_error: 16.1506\n",
            "Epoch 11/100\n",
            "35/35 [==============================] - 0s 735us/sample - loss: 0.0367 - mean_absolute_percentage_error: 38.9170 - val_loss: 0.0205 - val_mean_absolute_percentage_error: 22.4666\n",
            "Epoch 12/100\n",
            "35/35 [==============================] - 0s 719us/sample - loss: 0.0285 - mean_absolute_percentage_error: 38.1202 - val_loss: 0.0309 - val_mean_absolute_percentage_error: 28.5176\n",
            "Epoch 13/100\n",
            "35/35 [==============================] - 0s 727us/sample - loss: 0.0314 - mean_absolute_percentage_error: 38.2243 - val_loss: 0.0406 - val_mean_absolute_percentage_error: 31.7614\n",
            "Epoch 14/100\n",
            "35/35 [==============================] - 0s 702us/sample - loss: 0.0355 - mean_absolute_percentage_error: 37.9354 - val_loss: 0.0450 - val_mean_absolute_percentage_error: 32.1937\n",
            "Epoch 15/100\n",
            "35/35 [==============================] - 0s 699us/sample - loss: 0.0358 - mean_absolute_percentage_error: 36.5390 - val_loss: 0.0433 - val_mean_absolute_percentage_error: 30.6931\n",
            "Epoch 16/100\n",
            "35/35 [==============================] - 0s 631us/sample - loss: 0.0333 - mean_absolute_percentage_error: 35.0792 - val_loss: 0.0375 - val_mean_absolute_percentage_error: 28.4353\n",
            "Epoch 17/100\n",
            "35/35 [==============================] - 0s 626us/sample - loss: 0.0305 - mean_absolute_percentage_error: 34.9512 - val_loss: 0.0306 - val_mean_absolute_percentage_error: 26.3657\n",
            "Epoch 18/100\n",
            "35/35 [==============================] - 0s 634us/sample - loss: 0.0289 - mean_absolute_percentage_error: 35.9414 - val_loss: 0.0253 - val_mean_absolute_percentage_error: 24.9581\n",
            "Epoch 19/100\n",
            "35/35 [==============================] - 0s 659us/sample - loss: 0.0284 - mean_absolute_percentage_error: 37.1964 - val_loss: 0.0226 - val_mean_absolute_percentage_error: 24.2765\n",
            "Epoch 20/100\n",
            "35/35 [==============================] - 0s 682us/sample - loss: 0.0283 - mean_absolute_percentage_error: 37.8857 - val_loss: 0.0221 - val_mean_absolute_percentage_error: 24.1799\n",
            "Epoch 21/100\n",
            "35/35 [==============================] - 0s 725us/sample - loss: 0.0283 - mean_absolute_percentage_error: 37.7560 - val_loss: 0.0233 - val_mean_absolute_percentage_error: 24.4904\n",
            "Epoch 22/100\n",
            "35/35 [==============================] - 0s 754us/sample - loss: 0.0282 - mean_absolute_percentage_error: 37.1553 - val_loss: 0.0253 - val_mean_absolute_percentage_error: 25.0431\n",
            "Epoch 23/100\n",
            "35/35 [==============================] - 0s 646us/sample - loss: 0.0284 - mean_absolute_percentage_error: 36.6211 - val_loss: 0.0271 - val_mean_absolute_percentage_error: 25.6636\n",
            "Epoch 24/100\n",
            "35/35 [==============================] - 0s 604us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.4372 - val_loss: 0.0282 - val_mean_absolute_percentage_error: 26.1652\n",
            "Epoch 25/100\n",
            "35/35 [==============================] - 0s 721us/sample - loss: 0.0288 - mean_absolute_percentage_error: 36.5204 - val_loss: 0.0284 - val_mean_absolute_percentage_error: 26.4022\n",
            "Epoch 26/100\n",
            "35/35 [==============================] - 0s 824us/sample - loss: 0.0288 - mean_absolute_percentage_error: 36.6643 - val_loss: 0.0280 - val_mean_absolute_percentage_error: 26.3404\n",
            "Epoch 27/100\n",
            "35/35 [==============================] - 0s 869us/sample - loss: 0.0287 - mean_absolute_percentage_error: 36.7185 - val_loss: 0.0274 - val_mean_absolute_percentage_error: 26.0752\n",
            "Epoch 28/100\n",
            "35/35 [==============================] - 0s 1ms/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6859 - val_loss: 0.0269 - val_mean_absolute_percentage_error: 25.7720\n",
            "Epoch 29/100\n",
            "35/35 [==============================] - 0s 715us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.6579 - val_loss: 0.0265 - val_mean_absolute_percentage_error: 25.5705\n",
            "Epoch 30/100\n",
            "35/35 [==============================] - 0s 659us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.6927 - val_loss: 0.0263 - val_mean_absolute_percentage_error: 25.5202\n",
            "Epoch 31/100\n",
            "35/35 [==============================] - 0s 679us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.7627 - val_loss: 0.0263 - val_mean_absolute_percentage_error: 25.5831\n",
            "Epoch 32/100\n",
            "35/35 [==============================] - 0s 677us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.8006 - val_loss: 0.0265 - val_mean_absolute_percentage_error: 25.6859\n",
            "Epoch 33/100\n",
            "35/35 [==============================] - 0s 773us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.7741 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7713\n",
            "Epoch 34/100\n",
            "35/35 [==============================] - 0s 788us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.7127 - val_loss: 0.0270 - val_mean_absolute_percentage_error: 25.8174\n",
            "Epoch 35/100\n",
            "35/35 [==============================] - 0s 727us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6695 - val_loss: 0.0270 - val_mean_absolute_percentage_error: 25.8273\n",
            "Epoch 36/100\n",
            "35/35 [==============================] - 0s 678us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6699 - val_loss: 0.0270 - val_mean_absolute_percentage_error: 25.8115\n",
            "Epoch 37/100\n",
            "35/35 [==============================] - 0s 655us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6975 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7807\n",
            "Epoch 38/100\n",
            "35/35 [==============================] - 0s 688us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7206 - val_loss: 0.0267 - val_mean_absolute_percentage_error: 25.7473\n",
            "Epoch 39/100\n",
            "35/35 [==============================] - 0s 669us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.7241 - val_loss: 0.0267 - val_mean_absolute_percentage_error: 25.7243\n",
            "Epoch 40/100\n",
            "35/35 [==============================] - 0s 811us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.7156 - val_loss: 0.0267 - val_mean_absolute_percentage_error: 25.7198\n",
            "Epoch 41/100\n",
            "35/35 [==============================] - 0s 761us/sample - loss: 0.0285 - mean_absolute_percentage_error: 36.7096 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7316\n",
            "Epoch 42/100\n",
            "35/35 [==============================] - 0s 639us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7101 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7490\n",
            "Epoch 43/100\n",
            "35/35 [==============================] - 0s 689us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7116 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7608\n",
            "Epoch 44/100\n",
            "35/35 [==============================] - 0s 710us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7089 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7625\n",
            "Epoch 45/100\n",
            "35/35 [==============================] - 0s 648us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7040 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7570\n",
            "Epoch 46/100\n",
            "35/35 [==============================] - 0s 667us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7018 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7497\n",
            "Epoch 47/100\n",
            "35/35 [==============================] - 0s 676us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7040 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7444\n",
            "Epoch 48/100\n",
            "35/35 [==============================] - 0s 647us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7073 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7419\n",
            "Epoch 49/100\n",
            "35/35 [==============================] - 0s 648us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7081 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7416\n",
            "Epoch 50/100\n",
            "35/35 [==============================] - 0s 724us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7062 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7428\n",
            "Epoch 51/100\n",
            "35/35 [==============================] - 0s 748us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7037 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7443\n",
            "Epoch 52/100\n",
            "35/35 [==============================] - 0s 832us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7026 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7454\n",
            "Epoch 53/100\n",
            "35/35 [==============================] - 0s 761us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7026 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7450\n",
            "Epoch 54/100\n",
            "35/35 [==============================] - 0s 683us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7025 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7435\n",
            "Epoch 55/100\n",
            "35/35 [==============================] - 0s 678us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7019 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7415\n",
            "Epoch 56/100\n",
            "35/35 [==============================] - 0s 708us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7013 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7400\n",
            "Epoch 57/100\n",
            "35/35 [==============================] - 0s 679us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7010 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7394\n",
            "Epoch 58/100\n",
            "35/35 [==============================] - 0s 641us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7008 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7392\n",
            "Epoch 59/100\n",
            "35/35 [==============================] - 0s 701us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.7003 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7390\n",
            "Epoch 60/100\n",
            "35/35 [==============================] - 0s 713us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6995 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7387\n",
            "Epoch 61/100\n",
            "35/35 [==============================] - 0s 747us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6988 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7381\n",
            "Epoch 62/100\n",
            "35/35 [==============================] - 0s 685us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6983 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7374\n",
            "Epoch 63/100\n",
            "35/35 [==============================] - 0s 639us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6980 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7366\n",
            "Epoch 64/100\n",
            "35/35 [==============================] - 0s 701us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6976 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7358\n",
            "Epoch 65/100\n",
            "35/35 [==============================] - 0s 722us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6971 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7351\n",
            "Epoch 66/100\n",
            "35/35 [==============================] - 0s 689us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6965 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7346\n",
            "Epoch 67/100\n",
            "35/35 [==============================] - 0s 643us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6961 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7341\n",
            "Epoch 68/100\n",
            "35/35 [==============================] - 0s 673us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6956 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7336\n",
            "Epoch 69/100\n",
            "35/35 [==============================] - 0s 644us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6951 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7330\n",
            "Epoch 70/100\n",
            "35/35 [==============================] - 0s 705us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6946 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7323\n",
            "Epoch 71/100\n",
            "35/35 [==============================] - 0s 673us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6941 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7317\n",
            "Epoch 72/100\n",
            "35/35 [==============================] - 0s 680us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6937 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7311\n",
            "Epoch 73/100\n",
            "35/35 [==============================] - 0s 626us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6932 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7305\n",
            "Epoch 74/100\n",
            "35/35 [==============================] - 0s 801us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6927 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7300\n",
            "Epoch 75/100\n",
            "35/35 [==============================] - 0s 627us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6922 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7294\n",
            "Epoch 76/100\n",
            "35/35 [==============================] - 0s 658us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6918 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7288\n",
            "Epoch 77/100\n",
            "35/35 [==============================] - 0s 762us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6913 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7283\n",
            "Epoch 78/100\n",
            "35/35 [==============================] - 0s 757us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6908 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7277\n",
            "Epoch 79/100\n",
            "35/35 [==============================] - 0s 739us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6904 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7271\n",
            "Epoch 80/100\n",
            "35/35 [==============================] - 0s 764us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6899 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7266\n",
            "Epoch 81/100\n",
            "35/35 [==============================] - 0s 716us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6894 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7260\n",
            "Epoch 82/100\n",
            "35/35 [==============================] - 0s 757us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6890 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7255\n",
            "Epoch 83/100\n",
            "35/35 [==============================] - 0s 836us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6885 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7249\n",
            "Epoch 84/100\n",
            "35/35 [==============================] - 0s 754us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6881 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7244\n",
            "Epoch 85/100\n",
            "35/35 [==============================] - 0s 765us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6876 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7238\n",
            "Epoch 86/100\n",
            "35/35 [==============================] - 0s 741us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6872 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7233\n",
            "Epoch 87/100\n",
            "35/35 [==============================] - 0s 686us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6867 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7227\n",
            "Epoch 88/100\n",
            "35/35 [==============================] - 0s 828us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6862 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7222\n",
            "Epoch 89/100\n",
            "35/35 [==============================] - 0s 730us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6858 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7217\n",
            "Epoch 90/100\n",
            "35/35 [==============================] - 0s 711us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6853 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7211\n",
            "Epoch 91/100\n",
            "35/35 [==============================] - 0s 755us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6849 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7206\n",
            "Epoch 92/100\n",
            "35/35 [==============================] - 0s 799us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6844 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7201\n",
            "Epoch 93/100\n",
            "35/35 [==============================] - 0s 914us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6840 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7196\n",
            "Epoch 94/100\n",
            "35/35 [==============================] - 0s 733us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6836 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7190\n",
            "Epoch 95/100\n",
            "35/35 [==============================] - 0s 768us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6831 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7185\n",
            "Epoch 96/100\n",
            "35/35 [==============================] - 0s 755us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6827 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7180\n",
            "Epoch 97/100\n",
            "35/35 [==============================] - 0s 750us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6822 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7175\n",
            "Epoch 98/100\n",
            "35/35 [==============================] - 0s 1ms/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6818 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7170\n",
            "Epoch 99/100\n",
            "35/35 [==============================] - 0s 719us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6813 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7165\n",
            "Epoch 100/100\n",
            "35/35 [==============================] - 0s 779us/sample - loss: 0.0286 - mean_absolute_percentage_error: 36.6809 - val_loss: 0.0268 - val_mean_absolute_percentage_error: 25.7160\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f9faf1326d8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eRl3Gvu8ewD4",
        "colab_type": "text"
      },
      "source": [
        "# Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jhdu9lBie_t_",
        "colab_type": "text"
      },
      "source": [
        "There are several evaluation techniques to see how well our model performed. A common one for linear regression is mean squarred error."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BZl9Jz8qetzI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Predictions\n",
        "pred_train = model.predict(standardized_X_train)\n",
        "pred_test = model.predict(standardized_X_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UT30FvMUetwe",
        "colab_type": "code",
        "outputId": "24168a89-3644-4583-9a39-72b791f776be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((standardized_y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((standardized_y_test - pred_test) ** 2)\n",
        "print (f\"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}\")"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 0.03, test_MSE: 2.21\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TegkJM2-YKEq",
        "colab_type": "text"
      },
      "source": [
        "Since we only have one feature, it's easy to visually inspect the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I8nuCOjGeySz",
        "colab_type": "code",
        "outputId": "f3cca3ea-7428-4ce3-e3d6-8f5e0487f54f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 336
        }
      },
      "source": [
        "# Figure size\n",
        "plt.figure(figsize=(15,5))\n",
        "\n",
        "# Plot train data\n",
        "plt.subplot(1, 2, 1)\n",
        "plt.title(\"Train\")\n",
        "plt.scatter(standardized_X_train, standardized_y_train, label='y_train')\n",
        "plt.plot(standardized_X_train, pred_train, color='red', linewidth=1, linestyle='-', label='model')\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Plot test data\n",
        "plt.subplot(1, 2, 2)\n",
        "plt.title(\"Test\")\n",
        "plt.scatter(standardized_X_test, standardized_y_test, label='y_test')\n",
        "plt.plot(standardized_X_test, pred_test, color='red', linewidth=1, linestyle='-', label='model')\n",
        "plt.legend(loc='lower right')\n",
        "\n",
        "# Show plots\n",
        "plt.show()"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3YAAAE/CAYAAAAKZVEAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXhU9dn/8c9NDBDcooJLggi2GBVB\nolGrWGulGrRWAtW2FuvKQ9unPta2v1Sol+tjBaStWrWt1L1qXSFaQeOCFuuCIkHCYh4pBWUQWTQi\nOGCW7++PM8nMkMk6y5nl/bquXJzvmTNz7oO0d+75buacEwAAAAAgc/XyOwAAAAAAQHwo7AAAAAAg\nw1HYAQAAAECGo7ADAAAAgAxHYQcAAAAAGY7CDgAAAAAyHIUdkObMLM/MtprZIL9jAQAAQHqisAMS\nLFSEtfw0m1kwoj2hu5/nnGtyzu3mnPsgGfECAJBqic6VEZ/7ppmdl8hYgUyxi98BANnGObdby7GZ\nrZY00Tn3YnvXm9kuzrnGVMQGAEA66G6uBNA5euyAFDOzG8zsUTP7u5l9Luk8Mzs+9C1jvZl9ZGZ/\nNLP80PW7mJkzs8Gh9oOh1581s8/N7A0zG+LjIwEAkFChaQhXmdkqM9tkZg+ZWWHotV3N7BEz+ySU\nNxeY2V5m9ntJx0i6K9Tz93t/nwJILQo7wB/jJD0saU9Jj0pqlPRzSf0ljZI0RtKPO3j/DyVdJWlv\nSR9I+t9kBgsAQIr9P0mnSTpR0kBJDZJuDr02Ud6os2J5efNSSV86534l6W15vX+7hdpAzqCwA/zx\nL+fcP5xzzc65oHPubefcAudco3NulaSZkr7RwfufcM4tdM41SHpI0siURA0AQGr8RNJk59w659x2\nSddJ+r6Zmbwib4Ckr4Ty5tvOuW1+BgukA+bYAf74MLJhZodK+r2koyX1k/e/zQUdvH99xPEXknZr\n70IAADJJqHg7UNJcM3MRL/WStI+kuyXtL+kJM9tN0gOSrnLONaU8WCCN0GMH+MPt1L5T0lJJX3XO\n7SHpakmW8qgAAPCZc85JCkg6xTlXGPHT1zm3yTm3wzl3tXPuUEknSTpH0g9a3u5X3IDfKOyA9LC7\npM8kbTOzw9Tx/DoAALLdXyRNM7MDJcnM9jWz74SOv2Vmh5tZL0lb5M1Tbw6972NJB/sRMOA3Cjsg\nPfxK0gWSPpfXe/eov+EAAOCrmyS9KGleaAXp1yUdFXqtWNJT8nLmUklzFc6bN0s638w+NbObUhsy\n4C/zersBAAAAAJmKHjsAAAAAyHAUdgAAAACQ4SjsAAAAACDDUdgBAAAAQIajsAMAAACADLeL3wF0\npH///m7w4MF+hwEASLJ33nlnk3NugN9xZAryIwDkjq7myLQu7AYPHqyFCxf6HQYAIMnMbI3fMWQS\n8iMA5I6u5kiGYgIAAABAhqOwAwAAAIAMR2EHAAAAABmOwg4AAAAAMhyFHQAASWJm95jZBjNb2s7r\nJ5vZZ2a2OPRzdapjBABkh7ReFRMAgAx3n6TbJT3QwTWvOufOTE04AIBsRY8dAABJ4pybL+kTv+MA\nAGQ/CjsAAPx1vJm9a2bPmtkwv4MBAGQmhmICAOCfRZIOcs5tNbMzJFVJGhrrQjObJGmSJA0aNCh1\nEQIAuqyqJqAZ1XVaVx9UUWGBKstLVFFanJJ7U9gBALqtqiagW59ZorVbvtS+++ye0sSVTZxzWyKO\n55rZn8ysv3NuU4xrZ0qaKUllZWUuhWECALqgqiagKbNqFWxokiQF6oOaMqtWklKSIxmKCQDolqqa\ngBounqiXrz5Do1cuaE1cVTUBv0PLOGa2v5lZ6PhYeXl5s79RAQB6YkZ1XWtR1yLY0KQZ1XUpuT89\ndgCArnvvPVUcdZgk6Q8nTtBzJaMkhRMXvXbRzOzvkk6W1N/M1kq6RlK+JDnn/iLpbEk/NbNGSUFJ\nP3DO0RsHABloXX2wW+cTjcIOANA556RzzpGefFKSVPo/D+nTfntGXZKqxJVJnHPndvL67fK2QwAA\nZLiiwgIFYuTCosKClNyfoZgAgI4tWiT16uUVdbfcolFTX2pT1EmpS1wAAKSjyvISFeTnRZ0ryM9T\nZXlJSu6fkMLOzO4xsw1mtrSd1082s8/MbHHo5+pE3BcAkETOSaecIh19tNf+7DPp5z/3PXEBAJCO\nKkqLNXX8cBUXFsgkFRcWaOr44Rm3KuZ98oaSPNDBNa86585M0P0AAF3Q42WXX3tNOvFE7/iee6SL\nLmp9qeX9fi3nDABAuqooLfYtHyaksHPOzTezwYn4LABAYvRo2eWmJq+H7t13pX79pE2bpIK2Qyz9\nTFwAAKCtVM6xO97M3jWzZ81sWArvCwA5qdvLLldXS7vs4hV1jz8ubdsWs6gDAADpJ1WrYi6SdJBz\nbquZnSGpStLQWBea2SRJkyRp0KBBKQoPALJPl5ddbmiQvvIV6cMPpYEDpX//W+rdOwURAgCARElJ\nj51zbotzbmvoeK6kfDPr3861M51zZc65sgEDBqQiPABIe1U1AY2aNk9DJs/RqGnzurQZeHurVEad\nnzDBK+I+/FB69lnvT4o6AAAyTkoKOzPb38wsdHxs6L6bU3FvAMh0LXPlAvVBOYXnynVW3HW4euUn\nn0hm0sMPey80NkpjxiTpCQAAyCHTp0vf/37Kb5uo7Q7+LukNSSVmttbMLjGzn5jZT0KXnC1pqZm9\nK+mPkn7gnHOJuDcAZLtuz5ULaXfZ5V+eJ+2zj3fR1KnetgZ5eR1+FgAA6MT69d6XppMnS/vvn/Lb\nJ2pVzHM7ef12edshAAC6qctz5WKIWr0yEPDm0LVoavI2HgcAAPG54grpppu84/ffl7761ZSHQEYH\ngDTXpblynRk6NFzU3XWX10tHUQcAQHxWr/Z66W66SbrsMi+/+lDUSRR2AJD2Opwr15k33vASzsqV\nXru5WbrkkiRECQBAjrnkEmnIEO947Vrp1lt9DYfCDgDSXLtz5TrbINxMOuEE77hlLp23jhUAAOip\n5cu9fHrPPdL113v5tbiTnJwCqdrHDgAQh6i5cp35xz+ks84Kt1mrCgCA+Dnn5ddnnvHamzaFFyNL\nA/TYAUA2MQsXdffeS1EHAEAivPWWNzf9mWek22/38msaFXUSPXYAkB3uvluaODHcpqADACB+zc3S\nqFHSm2967S1bpN139zemdlDYAUA3VdUENKO6TuvqgyoqLFBleUnXh0kmQ+S8uTlzpDPO8C8WAACy\nxbx50ujR3vGDD0oTJvgbTyco7ACgG6pqApoyq7Z1w/BAfVBTZtVKUuqLu+uvl665Jtymlw4AgPg1\nNEiHHiqtWiX17y99+KHUt6/fUXWKOXYA0A0zqutai7oWwYYmzaiuS10QLatbthR1Cxb0uKirqglo\n1LR5GjJ5jkZNm6eqmkACAwUAIMPMni317u0Vdf/4h7RxY0YUdRI9dgDQLevqg906n3CTJkl//Wu4\nHUcvXVr1PgIA4KdgUNp3X2nrVq+3rrZW2iWzSiV67ACgG4oKC7p1PmEaGrxeupairq4u7qGXadH7\nCACA3+67T+rXzyvqXnlFWrEi44o6icIOALqlsrxEBfl5UecK8vNUWV6SvJv26+cNC2nhnHTIIXF/\nrO+9jwAA+GnLFu9L04sukk46SWpqkr7xDb+j6jEKOwDohorSYk0dP1zFhQUyScWFBZo6fnhyhi5u\n3eolnGCo0Fq1KqELpPjW+wgAgN9uuUXac0/v+J13pH/+09unLoNlXh8jAPisorQ4+XPQIrcwkJKy\n4mVleUnUHDspBb2PAAD4aflyadgw7/jss6XHHmubczMUhR0ApJP166UDDgi3N22S9tknKbdqKU7T\nak8+AACSJbKAe+cd6aij/IslCSjsACBdpKCXbmcp6X0EAMBPr78ujRoVbmfpvq+ZPZAUALLBBx9E\nF3Xbt2dt0gEAIKXMwkXdv/6V1fmVwg4A/GQmHXSQdzx+vJdw+vTxNyYAADLd009Hf2nqXHSvXRZi\nKCYA+KG2VhoxItxuasr41bgAAPCdc9H5dNky6fDD/YsnhfgtAgBSzSxc1F1+edskBAAAuu/uu8P5\n9MADvfyaI0WdRI8dAKTOq696G6C2aG7OmiWWEZuZ3SPpTEkbnHNHxHjdJN0q6QxJX0i60Dm3KLVR\nAkCGa26W8vLC7Q8/lAYO9C8en/AVMQCkglm4qPvd77xvESnqcsF9ksZ08PrpkoaGfiZJ+nMKYgKA\n7PHb34aLupNO8vJrDhZ1Ej12AJBcs2ZJ3/1uuJ3Fq3GhLefcfDMb3MElYyU94Jxzkt40s0IzO8A5\n91FKAgSATLVjh9S3b7i9ebO0996dvq2qJpC1+7fSYwcAyWIWLuoefJCiDrEUS/owor02dA4A0J6f\n/jRc1P3oR15+7WJRN2VWrQL1QTlJgfqgpsyqVVVNILnxpgg9dgCQaJWV3nDLFhR0SAAzmyRvuKYG\nDRrkczQA4IMtW6Q99wy3v/hCKijo8ttnVNcp2NAUdS7Y0KQZ1XVZ0WuXkB47M7vHzDaY2dJ2Xjcz\n+6OZrTSzJWZ2VCLuCwBpxyxc1FVXU9ShMwFJB0a0B4bOteGcm+mcK3POlQ0YMCAlwQFA2vj2t8NF\n3W9+4+XXbhR1krSuPtit85kmUT1290m6XdID7bweOTn8OHmTw49L0L0BwH8VFdJTT4XbFHTomqcl\nXWpmj8jLi58xvw4AIqxfLx1wQLjd0CDt0rMSpqiwQIEYRVxRYfcKxHSVkB4759x8SZ90cEnr5HDn\n3JuSCs3sgA6uB4DM0LK6ZUtR98ILFHVoZWZ/l/SGpBIzW2tml5jZT8zsJ6FL5kpaJWmlpL9K+m+f\nQgWA9DNsWLiou+02L7/2sKiTpMryEhXk50WdK8jPU2V5STxRpo1UzbFrb3I430oCyFzDhknLl4fb\ncRZ02bxSV65yzp3byetO0s9SFA4AZIaVK6WhQ8PtBO372pJTszXXpt3iKUwOB5D2vvxS6tMn3H73\nXWnEiLg+smWlrpZJ3S0rdUnKmoQDAECn+vb1tjKQpMcek845J6EfX1FanLV5NVXbHTA5HEB2MIsu\n6pyLu6iTOl6pCwCArLdwoZdjW4o65xJe1GW7VPXYMTkcQEbZeVjklFFFOvOkw8IXrFkjJXBUQbav\n1AUAQLsih1nOmyd985v+xZLBElLYhSaHnyypv5mtlXSNpHxJcs79Rd7k8DPkTQ7/QtJFibgvACTD\nzsMiX5syOvqCJCyOku0rdQEA0Mbzz0vl5eE2i4/FJSGFHZPDAWSTlmGRAz/7WP/6yyWt58dcXaXn\nrhublHtWlpdEFZNSdq3UBQBAlMheupoaaeRI/2LJEmm3eAoA+G1dfVCrp58ZdW7wFc/IdiTvntm+\nUhcAAJKkhx6SzjvPOy4slD791N94sgiFHYCMl9BtAt56S/+JKOpKfvmkduR7i6Uke1hkNq/UBQDI\ncc5JvSLWbVy1ShoyxL94slCqVsUEgKRomQ8XqA/KKbxNQFVNzIV3O2YmHXdca3PwFc+0FnUMiwQA\noId+//twUXf00V6RR1GXcPTYAchoHW0T0OXer6eekioqwu2mJlW9+5GKe9gLyEbjAABIamiQevcO\ntzdskNjOLGko7ABktLi3CYicvF1cLK1dK6nnwyLZaBwAAEm/+pX0hz94x+PHS08+6W88OYChmAAy\nWnvz3jqdD3fbbdFFnXOtRV082GgcAJDTtm3z8mtLUff55xR1KUJhByAjVNUENGraPA2ZPEejps1r\nnUNXWV6igvy8qGs7nQ9nJl12mXc8ZkxC981ho3EAQM76/vel3Xbzji+/3MuvLW0kHUMxAaS9rgxv\n7NKctssvl269Ndxmo3EAAOK3aVP03Lkvv5Ty8/2LJ0dR2AFIe50tkNKl+XCRwy5//nPplluSECkb\njQMAcszXviYtWOAd33STVFnpbzw5jMIOQNqLa3jj0KHSypXhdhJ66SKx0TgAICesXh29ZUFTU/Q+\ndUg5CjsAaa/Hwxsje+l++1vpN79JcGSxsdE4ACCrDRjgDb+UpAcekH70I3/jgSQKOwAZoNvDGyML\nOinpvXQAAOSEJUukI48Mt8mvaYX+UgAp097Klp2pKC3W1PHDVVxYIJNUXFigqeOHt+0Va2qKLuru\nv5+kAwBAIpiFi7pnnyW/piF67ACkRLwbd3c6vJFeOgAAEu+f/5ROPjncJr+mLXrsAKRE0jbubtkI\ntcXLL5N0AABIBLNwUbdgAfk1zdFjByAlkrJxN710AAAk3pNPSmef7R3n5UmNjf7Ggy6hxw5ASrS3\ngmWPNu7+6KPoou699yjqAACIl3Nefm0p6urqKOoyCIUdgJSoLC9RQX5e1LkebdxtJhUVhdvOSSXt\nf0ZPF2wBACCn/OlP4X3oSkq8/HrIIf7GhG5hKCaAlIh74+6lS6Xhw8PtDRu8fXQ6EO+CLQAAZL2m\nJmmXiJJg3TrpgAP8iwc9RmEHoNuqagI9KtB6vHF3D+fSdbRgC4UdACDnXXWVdMMN3nF5ufTcc/7G\ng7hQ2AHolpT2gr3wgnTaaeH2F19IBV2fk5eUBVsAAMh0waDUr1+4XV8v7bmnf/EgIZhjB6BbkrZt\nwc7Moos657pV1EkJXrAFAIBscPHF4aJu0iQvv1LUZQUKOwDdkvResJkzo4deNjX1eMXLhC3YAgBA\npquv9/Lrvfd67WBQuvNOf2NCQlHYAeiWpPaCmUk//nG47Vx4ha4eqCgt1tTxw1VcWCCTVFxYoKnj\nhzO/DgCQW0aPlvbayzu+7jovv/bt629MSDjm2AHolsrykqg5dlICesEqK6Xf/S7cTuCedD1esAUA\ngEy3dq104IHhdmOjt+E4slJCeuzMbIyZ1ZnZSjObHOP1C81so5ktDv1MTMR9AaRewnvBzMJF3ZFH\nstE4sg45EoAvDj44XNTdeaeXXynqslrcPXZmlifpDkmnSlor6W0ze9o5t3ynSx91zl0a7/0A+C8h\nvWDf/rY0d264TUGHLESOBJByK1ZIhx8ebjc3t902CFkpET12x0pa6Zxb5Zz7UtIjksYm4HMBZCuz\ncFF33nkUdchm5EgAqWMWLuqqqrz8SlGXMxJR2BVL+jCivTZ0bmffNbMlZvaEmR0Y43UA2a6oKDrB\nOCf97W/+xQMkHzkSQPK9+Wbb/DqW75ByTapWxfyHpMHOuRGSXpB0f3sXmtkkM1toZgs3btyYovAA\nJFXLN4YffeS1f/tbeumAsC7lSPIjgJjMpOOP945ffZX8msMSUdgFJEV+uzgwdK6Vc26zc25HqHmX\npKPb+zDn3EznXJlzrmzAgAEJCA+Ar8yityxwTvrNb/yLB0ithOVI8iOAKM8807aX7sQT/YsHvktE\nYfe2pKFmNsTMekv6gaSnIy8wswMimmdJWpGA+wJIZ42N0QnnkUf4FhG5iBwJILFaRsF85ztee+lS\n8iskJWBVTOdco5ldKqlaUp6ke5xzy8zsekkLnXNPS7rMzM6S1CjpE0kXxntfAOmpqiagiqMGRp8k\n4SBHkSMBJNS990oXX+wdFxd7+9QBIebS+BeusrIyt3DhQr/DANBFz8xfoTO/EV5iefyEGVox5Ij4\n9rlDTjCzd5xzZX7HkSnIj0COaW6O3oPugw+iNx5HVutqjkzV4ikAsp1ZVFE3+IpntGjgYQo2NGlG\ndZ2PgQEAkMGmTg0XdSee6I2CoahDDHEPxQSQ4/7zH+ngg1uboy/5s/7dPzrhrKsPdvoxVTUBzaiu\n07r6oIoKC1RZXkIvHwAgd335pdSnT7i9ebO0997+xYO0R48dgJiqagIaNW2ehkyeo1HT5qmqJtD2\nIrOoom7U1JfaFHWSVFRY0Om9psyqVaA+KCcpUB/UlFm1se8JAEC2u/TScFF33nleLx1FHTpBYQeg\njU4LrXnzole83LBBck6V5SUqyM+L+qyC/DxVlpd0eL8Z1XUKNjRFnWMIJwAg53z+uZdf77jDa2/b\nJv3tb/7GhIxBYQegjQ4LLTNp9OjwC85JoT21KkqLNXX8cBUXFsgkFRcWdGnhlPaGanZlCCcAAFnh\nrLOkPfbwjidP9vJrv37+xoSMwhw7AG3EKqh+sPg5Tau+PXziiy+kgrZDLCtKi7s9N66osECBGPfs\nbAgnAAAZ7+OPpf33D7cbGqRd+BUd3UePHYA2di6oVk8/M7qocy5mUddTPR3CCQBARjvyyHBRd+ut\nXn6lqEMPUdgBaKOl0Lr6xZlaPf3M1vNV73yYlM3GezqEEwCAjLRypTe1YckSr93cLF12mb8xIePx\nlQCANipKi1Vx1MCoc1WL1ia10OrJEE4AADLOrrt60xkk6dFHpe99z994kDUo7ABE+8pXpFWrwu1Q\nD12FT+EAAJAVFi2Sjj463E7CCBjkNgo7AGGRWxhIJB0AABIhMr++9JJ0yin+xYKsxRw7AF7CiUw6\nzlHUAQAQrxdeaJtfKeqQJPTYAWmiqiagGdV1WlcfVFFhgSrLS1Iz5ywy4RQVSYFA8u8JAEC2i8yv\nixZJpaX+xYKcQI8dkAaqagKaMqtWgfqgnKRAfVBTZtWqqiaJRVasXjqKOgAA4vP3v4fz6x57ePmV\nog4pQGEHpIEZ1XUKNjRFnQs2NGlGdV3ib+ZcdEH34x8z7BIAgHi15Ncf/tBr//vf0mef+RsTcgqF\nHZAG1tUHu3W+x8ykXhH/s3dO+stfEnsPAAByzS23hPNraamXXw8+2N+YkHMo7IA0UFRY0K3z3bZj\nR3Qv3e2300sHAEC8Ghu9/PqLX3jtjz/25tMBPqCwA9JAZXmJCvLzos4V5Oepsrwk/g83k/r2Dbed\nk372s/g/FwCAXPbrX0v5+d7xuHFeft13X39jQk5jVUwgDbSsfpnQVTE3boxOMNXV0mmntTZ9W4UT\nAIBM9sUX0q67httbtki77+5fPEAIhR2QJipKi9strLpdhHWy0XjLKpwtC7a0rMLZEgcAAIjh3HOl\nRx7xji+7TLr1Vn/jASJQ2AFprltF2LJl0hFHhNvLl0uHHdbmMztahZPCDgCAnWzeLPXvH27v2CH1\n7u1fPEAMzLED0lyXt0Iwiy7qnItZ1EkpXIUTAIBMd8IJ4aJu+nQvv1LUIQ3RYwekuU6LsFmzpO9+\nN/zCxo3R3yrGUFRYoECMz03YKpwAAGS6NWukwYPD7aam6C2DgDTDv04gzXW4FYJZdFHnXKdFnZTk\nVTgBAMh0++8fLuruv9/LrxR1SHP8CwXSXKwi7PIFj+m1KaPDJ7Zv79a+dBWlxZo6friKCwtkkooL\nCzR1/HDm1wEAclttrfel6ccfe23npPPP9zcmoIsSMhTTzMZIulVSnqS7nHPTdnq9j6QHJB0tabOk\n7zvnVifi3kC223krhP9MPzPq9apFazXj5te6vW1BR6twAgCQcyJXlJ47Vzr9dP9iAXog7h47M8uT\ndIek0yUdLulcMzt8p8sukfSpc+6rkm6WND3e+wK5pKK0WK+9dXt0UdfcrKpFazVlVq0C9UE5hVfM\nrKoJ+BYrAAAZZf786KLOOYo6ZKREDMU8VtJK59wq59yXkh6RNHana8ZKuj90/ISk0WY7b7QFoF1m\n0uzZ4bZzklnXV8wEAABtmUnf+IZ3/Oab3ZrWAKSbRBR2xZI+jGivDZ2LeY1zrlHSZ5L2ScC9gey2\n335tv0WMSDpsWwAAQA/Mnh3Or716ebn1uOP8jQmIU9otnmJmk8xsoZkt3Lhxo9/hAP4xkzZsCLdj\nfIvY4YqZANKCmY0xszozW2lmk2O83sfMHg29vsDMBqc+SiBHhEa8aPx4r/3ee942BkAWSERhF5B0\nYER7YOhczGvMbBdJe8pbRKUN59xM51yZc65swIABCQgPyDBmHfbSRWLbAiC9MQ8dSCN33hnesuCr\nX/Vyawn5EtkjEYXd25KGmtkQM+st6QeSnt7pmqclXRA6PlvSPOcYxAy0EVnQDR3a6Vh/ti0A0h7z\n0AG/NTV5+fUnP/HagYD0/vv+xgQkQdzbHTjnGs3sUknV8rY7uMc5t8zMrpe00Dn3tKS7Jf3NzFZK\n+kRe8Qegxc6/w3VzTzoKOSBtxZqHvvNEnqh56GbWMg99U0oiBLLZNddI11/vHZ92mlRd7W88QBIl\nZB8759xcSXN3Ond1xPF2Seck4l5AVnEuPCxEkn75S+n3v/cvHgBpy8wmSZokSYMGDfI5GiDNbd8u\nFUTMN6+vl/bc0794gBRIu8VTgJxhFl3UOUdRB2SfhM1DZw460EUTJ4aLuokTvfxKUYcckJAeOwDd\nEAxK/fqF23fdJV1yiX/xAEim1nno8gq4H0j64U7XtMxDf0PMQwd6rr5e2muvcDsYlPr29S8eIMXo\nsQNSySy6qHOOog7IYqG9W1vmoa+Q9FjLPHQzOyt02d2S9gnNQ/+lpDZbIgDoxKmnhou6a67x8itF\nHXIMPXZAKnz0kVRUFG6//LJ08sm+hQMgdZiHDiTRunVSccQCYo2NUl5e+9cDWYweOyDZzKKLOuco\n6gAAiNfQoeGi7i9/8fIrRR1yGD12yFlVNQHNqK7TuvqgigoLVFlekthtAxYvlkpLw+3/+z8vCQEA\ngJ6rq5MOPTTcbm5uu20QkIMo7JCTqmoCmjKrVsGGJklSoD6oKbNqJanT4q5LBWEc+9IBAIB2RObX\nWbOkceP8iwVIMwzFRE6aUV3XWtS1CDY0aUZ1XYfvaykIA/VBOYULwqqa0OrlTz0VnXQ+/ZSiDgCA\neC1YEJ1fnaOoA3ZCjx1y0rr6YLfOt+ioIKw4amD0xRR0AADEL7Kgmz9f+vrX/YsFSGP02CEnFRUW\ndOt8i1iF348XPKHXpowOn/jyS4o6AADiNXdu2146ijqgXfTYISdVlpdEzbGTpIL8PFWWl3T4vqLC\nAgUiirvV08+MvoCCDgCA+EUWdLW10hFH+BcLkCHosUNOqigt1tTxw1VcWCCTVFxYoKnjh3e6cEpl\neYkK8vN0yz9mRBV1Ve98SFEHAEC87r8/XNTtv7+XWynqgC6hxw45q6K0uNvbG1SUFreZS1e1aG1i\nt0kAACDXNDdH70G3Zo00aJB/8QAZiB47oKuOOqrtWH/nKOoAAIjHz34WLupOOMHLrxR1QLfRY4es\nlrBNyCMLuoMOklavTliMAIr/iJEAACAASURBVADkpB07pL59w+3166X99vMvHiDD0WOHrNXpnnNd\nYda2l46iDgCA+JxxRrioKy318itFHRAXCjtkrZ5uQt4qsqA7+2wWRwEAIF719V5+ffZZr71li7Ro\nkb8xAVmCwg5Zq6ebkMfspXv88QRGBgBADjrkEGmvvbzj737Xy6+77+5vTEAWobBD1ur2JuTNzdEF\n3fXX00sHAEC8AgEvv77/vtf+8kvpiSf8jQnIQhR2yFote85FancTcrPoZZadk666KskRAgCQ5cyk\ngaFtgv7f//Pya36+vzEBWYpVMZG1Wla/7HBVzG3bpN12C7cfe0w655wURwoAQJZZtix6Y/GdR8UA\nSDgKO2S1Djch3znBMOwSAID4RebXP/5R+p//8S8WIIcwFBO5Z+3a6KSzYAFFHQAA8Xr11baLj1HU\nASlDjx1yC710AAAkXmR+ZVoD4At67JAbFi2KTjpr1lDUAQAQryefbNtLR1EH+CKuHjsz21vSo5IG\nS1ot6XvOuU9jXNckqTbU/MA5d1Y89wW6hV46AAASLzK//vOf0kkn+RcLgLh77CZLesk5N1TSS6F2\nLEHn3MjQD0UdUmP27Oiks2ULRR0AAPG6/fa2vXQUdYDv4p1jN1bSyaHj+yW9IumKOD8TiB+9dAAA\nJJZzUq+IPoGlS6Vhw/yLB0CUeHvs9nPOfRQ6Xi9pv3au62tmC83sTTOriPOeQPumTYsu6hobKeoA\nAIjXr38dXdQ5R1EHpJlOe+zM7EVJ+8d46crIhnPOmVl7v0Ef5JwLmNnBkuaZWa1z7t/t3G+SpEmS\nNGjQoM7CA8LopQMAILEaGqTevcPtDz+UBg70Lx4A7eq0x8459y3n3BExfp6S9LGZHSBJoT83tPMZ\ngdCfq+QN1yzt4H4znXNlzrmyAQMG9OCRkHNOPbXtWH+KOgAA4vP974eLuq98xcutFHVA2op3jt3T\nki6QNC3051M7X2Bme0n6wjm3w8z6Sxol6aY47wt46KUDACCxtm6Vdt893P70U6mw0L94AHRJvHPs\npkk61czel/StUFtmVmZmd4WuOUzSQjN7V9LLkqY555bHeV/kul13pZcOAIBEO/rocFFXXu7lVoo6\nICPE1WPnnNssaXSM8wslTQwdvy5peDz3AaJEFnSFhd43iQAAoOc+/ljaP2JJhe3bpT59UnLrqpqA\nZlTXaV19UEWFBaosL1FFaXFK7g1kk3h77IDUMWvbS0dRByBNmdneZvaCmb0f+nOvdq5rMrPFoZ+n\nUx0noN12Cxd1//3fXn5NYVE3ZVatAvVBOUmB+qCmzKpVVU0gJfcHsgmFHTJDZEF3xhkMuwSQCSZL\nesk5N1TSS6F2LEHn3MjQz1mpCw857/33vfy6bZvXbmqS7rgjpSHMqK5TsKEp6lywoUkzqutSGgeQ\nDSjskN526qUbNfUlVd0w08eAAKDLxkq6P3R8vyT2cUX6MJMOOcQ7njq17ebjKbKuPtit8wDaF++q\nmEByNDZK+fmtzd+efLH+etx4KTREQxLj7wGku/2ccx+FjtdL2q+d6/qa2UJJjfIWGKtKSXTITW+9\nJR13XLjt8wiYosICBWIUcUWFBT5EA2Q2euyQFqpqAho1bZ6GTJ7jfYsYUdQNvuIZr6gLYYgGgHRh\nZi+a2dIYP2Mjr3POOUnt/QZ9kHOuTNIPJd1iZl9p516TzGyhmS3cuHFjYh8EucEsXNQ98IDvRZ0k\nVZaXqCA/L+pcQX6eKstLfIoIyFz02MF3LROn87Z+rv/c8r3W82/efI/OXb9vzPcwRANAOnDOfau9\n18zsYzM7wDn3kZkdIGlDO58RCP25ysxekVQq6d8xrpspaaYklZWV+f8bOTLHnDnSmWeG22lQ0LVo\nGX3DqphA/Cjs4LsZ1XVaccPpUecGX/GMircXqKhQDNEAkKmelnSBvD1eL5D01M4XhFbK/MI5t8PM\n+ksaJemmlEaJ7Ba5+Njzz0unnupfLO2oKC2mkAMSgKGY8Nfq1XptSngrxDMu/KMGX/GMJK9XjiEa\nADLYNEmnmtn7kr4VasvMyszsrtA1h0laaGbvSnpZ3hy75b5Ei+xy991ttwhKw6IOQOLQYwf/RCYc\nqbWga1FUWMAQDQAZyzm3WdLoGOcXSpoYOn5d0vAUh4ZstvPqlosWSaWl/sUDIGUo7JB6b74pHX98\na/O56oX6xasbpYh9bCJ75RiiAQBAF1x3nXTtteF2Gs2lA5B8FHZIrZ166eScxkjaPiBArxwAAD3R\n1CTtEvEr3apV0pAh/sUDwBcUdkiNRx6Rzj033N62TerXr7VJrxwAAD0wcaI3n06SBgyQNsRcfBVA\nDqCwQ/LF6KUDAABxCAajviDVxo1S//7+xQPAd6yKieS55prooq6piaIOAIB4ffOb4aJu1Cgvt1LU\nATmPHjt0W1VNF+bD0UsHAEBibd4cXcDtNK0BQG6jxw7dUlUT0JRZtQrUB+XkbR4+ZVatqmoC3gXj\nxrXdN4eiDgCA+BQVhYu6Cy7wcitFHYAI9NihW2ZU1ykYsS2BJAUbmjSjuk4VRw2MvpiCDgCA+Kxe\nHb3CZUND9AqYABDC/zOgW9bVB9uce/OO87X/1k/CJyjoAACIX+QImKuukq6/3r9YAKQ9Cjt0S1Fh\ngQIRxd3q6WeGXzz2WGnBgpTH1KU5fwAAZIp335VGjgy3m5vbzl0HgJ1Q2KFbKstLNGVWrVbccHrU\n+apFa30pplrm/LUMD22Z8yeJ4g4A4KseffEYWcDdeac0aVJygwSQNVg8Bd1SMbIoqqibddxZvhV1\nUsdz/gAA8Euni43t7KWX2i4+RlEHoBvosctiCR+iGGMLg/HxhRi3WHP+OjoPAEAqdLjYWEdbBD39\ntPSd76QgQgDZhh67LNXtbwo70tAQnXRmzkybBVKKCgu6dR4AgFTo0hePDz3UtpeOog5AD1HYZamE\nDVE0k3r3Dredk/7rvxIQYWJUlpeoID8v6lxBfp4qy0t8iggAgC588WgmnXeed/zGG2nzhSmAzEVh\nl6XiHqJYXx/9LeKLL6Zl0qkoLdbU8cNVXFggk1RcWKCp44ezcAoAwFftffF458cvt+2l+9rXUhwd\ngGwU1xw7MztH0rWSDpN0rHNuYTvXjZF0q6Q8SXc556bFc190budtCSLPdyrGXLp0VlFaTCEHAEgr\nLXmpZa578R599K8rTw1fUFcnHXKIT9EByEbx9tgtlTRe0vz2LjCzPEl3SDpd0uGSzjWzw+O8LzrR\noyGKq1ZFF3UrVqR9UQcAQLqqKC3Wa5NP0X++qA4XdX36eLmVog5AgsXVY+ecWyFJ1vGmmcdKWumc\nWxW69hFJYyUtj+fe6NjO3xR2uipmhvXSAQCQ9nbskPr2Dbc/+kjaf3//4gGQ1VKx3UGxpA8j2msl\nHZeC++a8Lg1RfP11adSocHvjRql//+QGBgBAtjvzTGnOHO94xAjp3Xf9jQdA1uu0sDOzFyXF+nrp\nSufcU4kOyMwmSZokSYMGDUr0xyMSvXQAACTWZ59JhYXh9pYt0u67+xcPgJzR6Rw759y3nHNHxPjp\nalEXkHRgRHtg6Fx795vpnCtzzpUNGDCgi7dAt/zjH9FF3fbtFHUAAMTr0EPDRd348V5upagDkCKp\nGIr5tqShZjZEXkH3A0k/TMF9EQu9dAAAJNa6dVJxxNSHHTui94AFgBSIa1VMMxtnZmslHS9pjplV\nh84XmdlcSXLONUq6VFK1pBWSHnPOLYsvbHTbbbdFF3XNzRR1AADEq1evcFH3q195uZWiDoAP4l0V\nc7ak2THOr5N0RkR7rqS58dwLcYgs6Hr39r5JBAAAPbdihXR4xO5Nzc1tR8UAQArFu48dUqCqJqBR\n0+ZpyOQ5GjVtnqpq2p2iGO2GG6KTjHMUdQAAxMssXNTdfLOXXynqAPgsFXPsEIeqmoCmzKpVsKFJ\nkhSoD2rKrFpJ6ngrg8gEM2GC9OCDyQwTAICUqqoJdH2v1kT517+kr3893GZKA4A0Qo9dmptRXdda\n1LUINjRpRnVd7DdcdFHbXjqKOgBAFmn50jNQH5RT+EvPLo9o6QmzcFH32GMUdQDSDoVdmltXH+z6\neTPpvvu84+uuI+kAALJSt7/0jMc997T9wvSccxJ/HwCIE0Mx01xRYYECMYq4osKCcOO446S33gq3\nu1DQ+TKEBQCABOjWl57xiCzoXnlF+sY3Evv5AJBA9NilucryEhXk50WdK8jPU2V5SXiydktRd999\nXS7qUj6EBQByjJmdY2bLzKzZzMo6uG6MmdWZ2Uozm5zKGDNV1JebXTjfbdde27aXjqIOQJqjsEtz\nFaXFmjp+uIoLC2SSigsLNHX8cFUcN8TbO6eFc9IFF3TpM1M6hAUActdSSeMlzW/vAjPLk3SHpNMl\nHS7pXDM7vL3r4enwS894tHxhet11Xvutt5jWACBjMBQzA1SUFoeHSTY0RG98+uKL0ujR3fq8lA1h\nAYAc5pxbIUnW8TL4x0pa6ZxbFbr2EUljJS1PeoAZrCUnJnRKwfnnS3/7W7jtnDdtYdo8pi0AyAgU\ndplk518OevgtYpfm7QEAUqFY0ocR7bWSjvMplowS9aVnPHb+wvTf/5YOPrjn2w0BgE8YipkJtm2L\nLupqa+MaGpK0ISwAkGPM7EUzWxrjZ2wS7jXJzBaa2cKNGzcm+uNz03HHRRd1zkkHHyyJaQsAMg89\ndukuQb10kZIyhAUAcpBz7ltxfkRA0oER7YGhc7HuNVPSTEkqKytj4lc8tmyR9twz3N64UerfP+oS\npi0AyDQUdulqwwZpv/3C7RhJJx4JG8ICAIjH25KGmtkQeQXdDyT90N+Qstwee0iff+4dDxkirVoV\n8zKmLQDINAzFTEdm0UWdcwkt6gAAyWdm48xsraTjJc0xs+rQ+SIzmytJzrlGSZdKqpa0QtJjzrll\nfsWc1dat8/JrS1H3xRftFnUS0xYAZB567NLJ++9LhxwSbm/bJvXr5188AIAec87NljQ7xvl1ks6I\naM+VNDeFoeWeyGkNp58uze38r5tpCwAyDYVduohMOvvsI23a5F8sAABkg2XLpCOOCLcbG6W8vPav\n3wnTFgBkEoZi+m3BguiirrGRog4AgHiZhYu6//kfb1pDN4o6AMg09NglUFVNoHtDNiILuhNPlF59\nNflBAgCQzf75T+nkk8Pt5ua2K0wDQBaixy5BWjYyDdQH5RTeyLSqJsaq1U8/HZ1kmpsp6gAAiJdZ\nuKj7wx+8XjqKOgA5gsIuQbq8kamZNDa0b+1FF5F0AACI1yOPROdS56Rf/MK/eADABwzFTJBONzJ9\n6impoiL8QgI2GgcAIOdFFnSPPip973v+xQIAPqKw66b25tF1uJFpZNK5/37p/PNTGDEAAFno6afD\nI2AkvjAFkPMo7LqhZR5dy5DLlnl0kreRaeRrknR+7fO6fu4fwx9A0gEAID7OSb0iZpIsXy4ddph/\n8QBAmmCOXTd0NI+uorRYU8cPV3FhgUzS6ulnhou6Z5+lqAMAIF533x0u6gYP9nIrRR0ASKLHrls6\nm0dXUVqsirpXpXPPDb9IQQcAQHyam6P3oFu7Vipm43AAiESPXTcUFRa0f75ldcuWou6ttyjqAACI\n1w03hIu6U07xcitFHQC0EVdhZ2bnmNkyM2s2s7IOrlttZrVmttjMFsZzTz9VlpeoID8v6lxBfp7u\n+Pzt8NCQYcO8pHPMMT5ECABAltixw/vC9KqrvPYnn0gvveRvTACQxuIdirlU0nhJd3bh2m865zbF\neT9fVZR63xC2rIo5cI/eevXK08IXrFsnHXCAT9EBAJAlfvpT6S9/8Y4vuki65x5/4wGADBBXYeec\nWyFJlkMbbFeUFnsF3v/+r3Tl1d7JU0+Vnn/e38AAAMh0W7ZIe+4Zbn/xhVQQexoEACBaqubYOUnP\nm9k7ZjYpRfdMjpahIVeHirpPP6WoAwAgXt/+driou+oqb1oDRR0AdFmnPXZm9qKk/WO8dKVz7qku\n3udE51zAzPaV9IKZveecm9/O/SZJmiRJgwYN6uLHp8ill0p33OEdX3ihdO+9voYDAEDGW78+ehpD\nY2P0CpgAgC7ptLBzzn0r3ps45wKhPzeY2WxJx0qKWdg552ZKmilJZWVl6bGs5Nat0u67h9sMDQEA\nIH4/+5n0pz95x3fcIf33f/sbDwBksKQPxTSzXc1s95ZjSafJW3QlM5x9driomzyZoSEAAMRr82Zv\nWkNLUdfcTFEHAHGKa/EUMxsn6TZJAyTNMbPFzrlyMyuSdJdz7gxJ+0maHVpgZRdJDzvnnosz7uTb\ntEkaMCDcbmiQdolvEdGqmkDrippFhQWqLC9pXWkTAICccP310jXXeMfLl0uHHeZvPACQJeJdFXO2\npNkxzq+TdEboeJWkI+O5T8odf7z05pve8R/+IP3iF3F/ZFVNQFNm1SrY0CRJCtQHNWVWrSRR3AEA\nsl8gIA0c6B1fcol0113+xgMAWSbefeyyywcfSAcdFG43N3tDRRJgRnVda1HXItjQpBnVdRR2AIDs\ndvnl0q23esf/+Y80eLCv4QBANkrVdgfpb+DAcFH3wAPeXLoE7s+3rj7YrfMAAGS8lSu9XHrrrdIV\nV3i5laIOAJKCHrvly6Vhw8Jtl5yFOIsKCxSIUcQVFbIQCwAgC02YID38sHe8fr20337+xgMAWS5n\ne+yqagJav9d+rUXdG7fen7SiTpIqy0tUkB+9L09Bfp4qy0uSdk8AAFJuyRKvl+7hh6WbbvJyK0Ud\nACRdTvbYPffc26o4/djW9uArnlHBxjxNrQkkbb5by+eyKiaQOxoaGrR27Vpt377d71DSRt++fTVw\n4EDl5+f7HQoSzTnptNOkF1/02p9+KhUW+hsTgIxAvvTEmyNzr7C77DKNue02SdIJP71H6/bYV1Jq\nFjKpKC2mkANyyNq1a7X77rtr8ODBsgTO2c1Uzjlt3rxZa9eu1ZAhQ/wOB4n0+uvSqFHe8V//Kk2c\n6G88ADIK+TIxOTJ3CruVK6WhQyVJdxz/Pc046fw2l7CQCYBE2r59e04nqZ2ZmfbZZx9t3LjR71CQ\nKE1NUlmZtHix1Levt/F4v35+RwUgw5AvE5Mjc2OO3Y9+1FrUaf16PXzWj2NexkImABItl5NULPx9\nZJHqammXXbyi7vHHpWCQog5Aj5Ef4v87yP7Cbu+9pQcflKZPb53AzUImANB9gwcP1qZNm+K+Bhnu\nyy+l4mJpzBhvq6AdO6Szz/Y7KgBIC37myuwv7KqrpU8+kX7969ZTFaXFmjp+uIoLC2SSigsLNHX8\ncOa/AQASxszOMbNlZtZsZmUdXLfazGrNbLGZLUxljN32+ONSnz7SunXSc89JH34o9e7td1QAAOVC\nYXfMMdJee7U5XVFarNcmn6L/TPu2Xpt8CkUdgKy0evVqHXroobrwwgt1yCGHaMKECXrxxRc1atQo\nDR06VG+99ZY++eQTVVRUaMSIEfra176mJUuWSJI2b96s0047TcOGDdPEiRPlIraEefDBB3Xsscdq\n5MiR+vGPf6ympia/HjGdLZU0XtL8Llz7TefcSOdcuwWgr7Zt8wq4731POvpoqbFRKi/3OyoASIhs\nyZXZX9gBQI5buXKlfvWrX+m9997Te++9p4cfflj/+te/9Lvf/U433nijrrnmGpWWlmrJkiW68cYb\ndf753uJS1113nU488UQtW7ZM48aN0wcffCBJWrFihR599FG99tprWrx4sfLy8vTQQw/5+YhpyTm3\nwjlX53cccZs5U9ptN6mhwVv9cuFCKS+v8/cBQAbJhlyZO6tiAoDfkjExPOKbwfYMGTJEw4cPlyQN\nGzZMo0ePlplp+PDhWr16tdasWaMnn3xSknTKKado8+bN2rJli+bPn69Zs2ZJkr797W9rr9Doh5de\neknvvPOOjjnmGElSMBjUvvvum/hnyx1O0vNm5iTd6Zyb6XdAkrx96Pbe2zseM0aaOzeh/4aragLs\n7QogNh/yZTbkSgo7AEiVLhRhydCnT5/W4169erW2e/XqpcbGxm5vhOqc0wUXXKCpU6cmNM5MZGYv\nSto/xktXOuee6uLHnOicC5jZvpJeMLP3nHNthm+a2SRJkyRp0KBBPY65S6ZPlyZP9o6XLJFCv+wk\nSlVNQFNm1SrY4A1LCtQHNWVWrSRR3AHwJV9mQ65kKCYA5Livf/3rrcNDXnnlFfXv31977LGHTjrp\nJD388MOSpGeffVaffvqpJGn06NF64okntGHDBknSJ598ojVr1vgTvM+cc99yzh0R46erRZ2cc4HQ\nnxskzZZ0bDvXzXTOlTnnygYMGJCYB9jZ+vXeN+WTJ0sTJni/XCW4qJOkGdV1rUVdi2BDk2ZUZ/7I\nVQDZKRNyJT12AJDjrr32Wl188cUaMWKE+vXrp/vvv1+SdM011+jcc8/VsGHDdMIJJ7T2Eh1++OG6\n4YYbdNppp6m5uVn5+fm64447dNBBB/n5GBnJzHaV1Ms593no+DRJ1/sSzBVXSDfd5B2vXCl95StJ\nu9W6+mC3zgOA3zIhV5rzaWhQV5SVlbmFC9N75WcAaM+KFSt02GGH+R1G2on192Jm76TtipA9ZGbj\nJN0maYCkekmLnXPlZlYk6S7n3BlmdrC8XjrJ+7L1Yefcbzv77ITmx9WrpSFDvOPLL5duvjkxn9uB\nUdPmKRCjiCsuLNBrk09J+v0BpBfyZVg8OZIeOwAAksA5N1vhoi3y/DpJZ4SOV0k6MsWhhV1yiXTP\nPd5xICAVFaXktpXlJVFz7CSpID9PleUlKbk/AGQj5tgBAJCLtmzxirobbvDm0qWoqJO8BVKmjh+u\n4sICmbyeuqnjh7NwCgDEgR47AABy0R57+LZSq+QVdxRyAJA49NgBAAAAQIajsAMAAACADEdhBwAA\nAAAZjsIOAAAAADqxevXq1s3Ie+LGG29MYDRtUdgBQI6LJ1GdcMIJCY4GAID0lNWFnZnNMLP3zGyJ\nmc02s8J2rhtjZnVmttLMJsdzTwDIVlU1AY2aNk9DJs/RqGnzVFUTSMl9O0pUjY2NHb739ddfT0ZI\nAAC0K9H58uqrr9Ytt9zS2r7yyit16623trlu8uTJevXVVzVy5EjdfPPNampqUmVlpY455hiNGDFC\nd955pyTpo48+0kknnaSRI0fqiCOO0KuvvqrJkycrGAxq5MiRmjBhQlzxtife7Q5ekDTFOddoZtMl\nTZF0ReQFZpYn6Q5Jp0paK+ltM3vaObc8znsDQNaoqglEbdgcqA9qyqxaSerxkvBXX3219t57b11+\n+eWSvES177776uc//3nUdZMnT9aKFSs0cuRIXXDBBdprr700a9Ysbd26VU1NTZozZ47Gjh2rTz/9\nVA0NDbrhhhs0duxYSdJuu+2mrVu36pVXXtG1116r/v37a+nSpTr66KP14IMPysx6+lcCAEAbyciX\nF198scaPH6/LL79czc3NeuSRR/TWW2+1uW7atGn63e9+p2eeeUaSNHPmTO255556++23tWPHDo0a\nNUqnnXaaZs2apfLycl155ZVqamrSF198oa9//eu6/fbbtXjx4h4+eefiKuycc89HNN+UdHaMy46V\ntNI5t0qSzOwRSWMlJb2wq6oJaEZ1ndbVB1VUWKDK8hL2zAGQlmZU17UmqRbBhibNqK5LeaK67777\ntGjRIi1ZskR77723GhsbNXv2bO2xxx7atGmTvva1r+mss85qU7TV1NRo2bJlKioq0qhRo/Taa6/p\nxBNP7FHsSC7yI4BMlYx8OXjwYO2zzz6qqanRxx9/rNLSUu2zzz6dvu/555/XkiVL9MQTT0iSPvvs\nM73//vs65phjdPHFF6uhoUEVFRUaOXJkj+LqrkRuUH6xpEdjnC+W9GFEe62k4xJ435iSUc0DQLKs\nqw9263xX9DRRSdKpp56qvffeW5LknNNvfvMbzZ8/X7169VIgENDHH3+s/fffP+o9xx57rAYOHChJ\nGjlypFavXk1hl4bIjwAyWTLypSRNnDhR9913n9avX6+LL764S+9xzum2225TeXl5m9fmz5+vOXPm\n6MILL9Qvf/lLnX/++XHF1xWdzrEzsxfNbGmMn7ER11wpqVHSQ/EGZGaTzGyhmS3cuHFjjz+no2oe\nANJNUWFBt853VUuiuvfee7ucqCRp1113bT1+6KGHtHHjRr3zzjtavHix9ttvP23fvr3Ne/r06dN6\nnJeX1+n8PPiD/AggkyUrX44bN07PPfec3n777ZiFmiTtvvvu+vzzz1vb5eXl+vOf/6yGhgZJ0v/9\n3/9p27ZtWrNmjfbbbz/913/9lyZOnKhFixZJkvLz81uvTYZOe+ycc9/q6HUzu1DSmZJGO+dcjEsC\nkg6MaA8MnWvvfjMlzZSksrKyWJ/XJcmq5gEgGSrLS6J6USSpID9PleUlcX3uuHHjdPXVV6uhoaHd\nBVJ2TlQ7++yzz7TvvvsqPz9fL7/8stasWRNXTPAX+RFAJktWvuzdu7e++c1vqrCwUHl5eTGvGTFi\nhPLy8nTkkUfqwgsv1M9//nOtXr1aRx11lJxzGjBggKqqqvTKK69oxowZys/P12677aYHHnhAkjRp\n0iSNGDFCRx11lB56KO7+sDbiGoppZmMk/VrSN5xzX7Rz2duShprZEHkF3Q8k/TCe+3ZFUWGBAjGS\nVLzVPAAkQ8sQuETPe+pJotprr72iXp8wYYK+853vaPjw4SorK9Ohhx4aV0zwF/kRQCZLVr5sbm7W\nm2++qccff7zda/Lz8zVv3ryoczfeeGObbQwuuOACXXDBBW3eP336dE2fPj2uODsS7xy72yX1kfRC\naBL9m865n5hZkaS7nHNnhFbMvFRStaQ8Sfc455bFed9OJauaB4BkqSgtTvgcp54mqgsvvLD1uH//\n/nrjjTdivnfr1q2SpJNPPlknn3xy6/nbb7+950EjqciPADJdovPl8uXLdeaZZ2rcuHEaOnRowj43\n1eJdFfOr7ZxfJ+mMq1QEfwAABwFJREFUiPZcSXPjuVd3JauaB4BMkS2JColFfgSAaIcffrhWrVrV\n2q6trdWPfvSjqGv69OmjBQsWpDq0bknkqphpJxnffgNApsiWRIXEIz8CQPuGDx+e1P3mkiWrCzsA\nQFimJioAANC5Trc7AAD0XOzFgnMXfx8AgFjID/H/HVDYAUCS9O3bV5s3byZZhTjntHnzZvXt29fv\nUAAAaYR8mZgcyVBMAEiSgQMHau3atdq4caPfoaSNvn37auDAgX6HAQBII+RLT7w5ksIOAJIkPz9f\nQ4YM8TsMAADSGvkyMRiKCQAAAAAZjsIOAAAAADIchR0AAAAAZDhL59VnzGyjpDV+x9FF/SVt8juI\nBMiW55Cy51my5Tmk7HkWniPxDnLODfA7iEzhc35Mp383yZILzyjlxnPmwjNKPGc2ifWMXcqRaV3Y\nZRIzW+icK/M7jnhly3NI2fMs2fIcUvY8C8+BXJYL/25y4Rml3HjOXHhGiefMJvE8I0MxAQAAACDD\nUdgBAAAAQIajsEucmX4HkCDZ8hxS9jxLtjyHlD3PwnMgl+XCv5tceEYpN54zF55R4jmzSY+fkTl2\nAAAAAJDh6LEDAAAAgAxHYddDZnaOmS0zs2Yza3flGjNbbWa1ZrbYzBamMsau6MZzjDGzOjNbaWaT\nUxljV5nZ3mb2gpm9H/pzr3auawr991hsZk+nOs72dPZ3bGZ9zOzR0OsLzGxw6qPsXBee40Iz2xjx\n32CiH3F2xszuMbMNZra0ndfNzP4Yes4lZnZUqmPsii48x8lm9lnEf4+rUx0j0lu25LuOZFMu7Eim\n58mOZEsO7Uy25NiOZEv+7UiycjOFXc8tlTRe0vwuXPtN59zINF2etdPnMLM8SXdIOl3S4ZLONbPD\nUxNet0yW9JJzbqikl0LtWIKh/x4jnXNnpS689nXx7/gSSZ86574q6WZJ01MbZee68W/l0Yj/Bnel\nNMiuu0/SmA5eP13S0NDPJEl/TkFMPXGfOn4OSXo14r/H9SmICZklW/JdR7IpF3YkY/NkR7Ilh3Ym\ny3JsR+5TduTfjtynJORmCrsecs6tcM7V+R1HvLr4HMdKWumcW+Wc+1LSI5LGJj+6bvv/7d1LixxV\nGMbx/wNBAyJeEkziBXFAEFwJQTS6UnGRRaLowpVZBDQLv4E7N4IfwFU2cZOFATVCRA1RXA3eYBi8\noMaNGcYEFSJugsLros5Io/apmp7uqjqnnx8Uc6q6p3nf6i6eOd3VNUeBU2l8CnhqwFq2q8s+nuzv\nDPC4JPVYYxelvFZaRcQnwG+ZuxwF3ojGKnCzpAP9VNddhz7MsmrJu5zKsjCn5JzMqSVD29TwGmxV\nS/7mLCqbPbFbvAA+kPSFpBeGLmZGdwA/TaxfStvGZl9EbKbxz8C+KffbLelzSauSxhJqXfbxP/eJ\niL+Aq8CeXqrrrutr5Zl0+sQZSXf1U9rclXJcdPGwpDVJ70m6f+hirFg15F1ODcd8yTmZU0uGtlmm\njM2p4VjsYtvZvGvRFZVM0nlg///c9HJEvNPxYR6NiA1JtwEfSvo2zdJ7M6c+RiHXy+RKRISkaZd8\nvTs9JyvABUnrEXFx3rXaVO8CpyPimqQXad5BfWzgmpbZlzTHxB+SDgNv05zeYkuklrzLqSkLc5yT\nS88ZW4eZstkTu4yIeGIOj7GRfl6R9BbNx+i9Bt0c+tgAJt/xuTNt612uF0mXJR2IiM30kfyVKY+x\n9Zz8KOlj4AFg6MDqso+37nNJ0i7gJuDXfsrrrLWPiJis+STwWg91LcJojoudiIjfJ8bnJL0uaW9E\n/DJkXdavWvIup6YszKk4J3NqydA2y5SxOUUcizsxazb7VMwFknSDpBu3xsCTNF/QLs1nwL2S7pF0\nHfAcMMarZJ0FjqXxMeA/78BKukXS9Wm8F3gE+Lq3Cqfrso8n+3sWuBDj+0eUrX386zz4I8A3PdY3\nT2eB59PVuR4Crk6c4lQMSfu3vmci6UGaXCjtjx0bWEV5l1NKFuaUnJM5tWRom2XK2Jwq8jdn5myO\nCC8zLMDTNOf0XgMuA++n7bcD59J4BVhLy1c0p3sMXvt2+0jrh4HvaN6xG10fqcY9NFf5+h44D9ya\nth8ETqbxIWA9PSfrwPGh687tY+AV4Ega7wbeBH4APgVWhq55xj5eTcfDGvARcN/QNU/p4zSwCfyZ\njpHjwAngRLpdNFcnu5heSweHrnnGPl6aeD5WgUND1+xlXEstebfTHtP66LOwpc+ic7KltyoydA59\nFpGxLT1Wkb877HGmbFb6ZTMzMzMzMyuUT8U0MzMzMzMrnCd2ZmZmZmZmhfPEzszMzMzMrHCe2JmZ\nmZmZmRXOEzszMzMzM7PCeWJnZmZmZmZWOE/szMzMzMzMCueJnZmZmZmZWeH+Bmud9cGf2dVAAAAA\nAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 1080x360 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T621buxbe0VQ",
        "colab_type": "text"
      },
      "source": [
        "# Inference"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o4W84oQzfNn_",
        "colab_type": "text"
      },
      "source": [
        "After training a model, we can use it to predict on new data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O_ZKW-p2e0fN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Feed in your own inputs\n",
        "sample_indices = [10, 15, 25]\n",
        "X_infer = np.array(sample_indices, dtype=np.float32)\n",
        "standardized_X_infer = X_scaler.transform(X_infer.reshape(-1, 1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ZjQ54zKazbE",
        "colab_type": "text"
      },
      "source": [
        "Recall that we need to unstandardize our predictions.\n",
        "\n",
        "$ \\hat{y}_{scaled} = \\frac{\\hat{y} - \\mu_{\\hat{y}}}{\\sigma_{\\hat{y}}} $\n",
        "\n",
        "$ \\hat{y} = \\hat{y}_{scaled} * \\sigma_{\\hat{y}} + \\mu_{\\hat{y}} $"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PMAdz712aE34",
        "colab_type": "code",
        "outputId": "aed82cf8-86fe-4fc0-8971-884b8ca27b52",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Unstandardize predictions\n",
        "pred_infer = model.predict(standardized_X_infer) * np.sqrt(y_scaler.var_) + y_scaler.mean_\n",
        "for i, index in enumerate(sample_indices):\n",
        "    print (f\"{df.iloc[index]['y']:.2f} (actual) → {pred_infer[i][0]:.2f} (predicted)\")"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "35.73 (actual) → 43.49 (predicted)\n",
            "59.34 (actual) → 60.03 (predicted)\n",
            "97.04 (actual) → 93.09 (predicted)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H5CB4zRFe37l",
        "colab_type": "text"
      },
      "source": [
        "# Interpretability"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OhXo8CbPBZ-G",
        "colab_type": "text"
      },
      "source": [
        "Linear regression offers the great advantage of being highly interpretable. Each feature has a coefficient which signifies its importance/impact on the output variable y. We can interpret our coefficient as follows: by increasing X by 1 unit, we increase y by $W$ (~3.65) units. \n",
        "\n",
        "**Note**: Since we standardized our inputs and outputs for gradient descent, we need to apply an operation to our coefficients and intercept to interpret them. See proof in the `From scratch` section above."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lJOOjjLze6_U",
        "colab_type": "code",
        "outputId": "2a735554-e3cd-42bc-db32-c83b0cc8ea8f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Unstandardize coefficients (proof is in the `From Scratch` section above)\n",
        "W = model.layers[0].get_weights()[0][0][0]\n",
        "b = model.layers[0].get_weights()[1][0]\n",
        "W_unscaled = W * (y_scaler.scale_/X_scaler.scale_)\n",
        "b_unscaled = b * y_scaler.scale_ + y_scaler.mean_ - np.sum(W_unscaled*X_scaler.mean_)\n",
        "print (\"[actual] y = 3.5X + noise\")\n",
        "print (f\"[model] y_hat = {W_unscaled[0]:.1f}X + {b_unscaled[0]:.1f}\") "
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[actual] y = 3.5X + noise\n",
            "[model] y_hat = 3.3X + 10.4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rToCXKqeJcvj",
        "colab_type": "text"
      },
      "source": [
        "# Regularization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4GFv8xRJmOZ",
        "colab_type": "text"
      },
      "source": [
        "Regularization helps decrease overfitting. Below is L2 regularization (ridge regression). There are many forms of regularization but they all work to reduce overfitting in our models. With L2 regularization, we are penalizing the weights with large magnitudes by decaying them. Having certain weights with high magnitudes will lead to preferential bias with the inputs and we want the model to work with all the inputs and not just a select few. There are also other types of regularization like L1 (lasso regression) which is useful for creating sparse models where some feature cofficients are zeroed out, or elastic which combines L1 and L2 penalties. \n",
        "\n",
        "**Note**: Regularization is not just for linear regression. You can use it to regularize any model's weights including the ones we will look at in future lessons."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_OcpRxF-Oj7",
        "colab_type": "text"
      },
      "source": [
        "$ J(\\theta) = = \\frac{1}{2}\\sum_{i}(X_iW - y_i)^2 + \\frac{\\lambda}{2}W^TW$\n",
        "\n",
        "$ \\frac{\\partial{J}}{\\partial{W}}  = X (\\hat{y} - y) + \\lambda W $\n",
        "\n",
        "$W = W- \\alpha\\frac{\\partial{J}}{\\partial{W}}$\n",
        "* $\\lambda$ is the regularzation coefficient"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HHaazL9f8QZX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.regularizers import l2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VTIUZLbGZP4e",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "L2_LAMBDA = 1e-2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ORwkUqcuZhbX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Linear model with L2 regularization\n",
        "class LinearRegressionL2Regularization(Model):\n",
        "    def __init__(self, hidden_dim):\n",
        "        super(LinearRegressionL2Regularization, self).__init__()\n",
        "        self.fc1 = Dense(units=hidden_dim, activation='linear',\n",
        "                        kernel_regularizer=l2(l=L2_LAMBDA))\n",
        "        \n",
        "    def call(self, x_in, training=False):\n",
        "        \"\"\"Forward pass.\"\"\"\n",
        "        y_pred = self.fc1(x_in)\n",
        "        return y_pred\n",
        "    \n",
        "    def sample(self, input_shape):\n",
        "        x_in = Input(shape=input_shape)\n",
        "        return Model(inputs=x_in, outputs=self.call(x_in)).summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IWCvYxBxZhd5",
        "colab_type": "code",
        "outputId": "3670f885-252f-49fd-afc1-8c64a9328003",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "# Initialize the model\n",
        "model = LinearRegressionL2Regularization(hidden_dim=HIDDEN_DIM)\n",
        "model.sample(input_shape=(INPUT_DIM,))"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_3 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 1)                 2         \n",
            "=================================================================\n",
            "Total params: 2\n",
            "Trainable params: 2\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DJ-XpSYAoNBX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compile\n",
        "model.compile(optimizer=Adam(lr=LEARNING_RATE),\n",
        "              loss=MeanSquaredError(),\n",
        "              metrics=[MeanAbsolutePercentageError()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xlt9IuaCoM-X",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "f771fbe4-a860-42cc-8550-a6f7d1e42a2f"
      },
      "source": [
        "# Training\n",
        "model.fit(x=standardized_X_train, \n",
        "          y=standardized_y_train,\n",
        "          validation_data=(standardized_X_val, standardized_y_val),\n",
        "          epochs=NUM_EPOCHS,\n",
        "          batch_size=BATCH_SIZE,\n",
        "          shuffle=SHUFFLE,\n",
        "          verbose=1)"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 35 samples, validate on 7 samples\n",
            "Epoch 1/100\n",
            "35/35 [==============================] - 0s 12ms/sample - loss: 0.3207 - mean_absolute_percentage_error: 63.4316 - val_loss: 0.1373 - val_mean_absolute_percentage_error: 47.1876\n",
            "Epoch 2/100\n",
            "35/35 [==============================] - 0s 681us/sample - loss: 0.0765 - mean_absolute_percentage_error: 56.2515 - val_loss: 0.0292 - val_mean_absolute_percentage_error: 25.8445\n",
            "Epoch 3/100\n",
            "35/35 [==============================] - 0s 763us/sample - loss: 0.0624 - mean_absolute_percentage_error: 46.6229 - val_loss: 0.0400 - val_mean_absolute_percentage_error: 17.3323\n",
            "Epoch 4/100\n",
            "35/35 [==============================] - 0s 764us/sample - loss: 0.0910 - mean_absolute_percentage_error: 46.5634 - val_loss: 0.0335 - val_mean_absolute_percentage_error: 16.3579\n",
            "Epoch 5/100\n",
            "35/35 [==============================] - 0s 768us/sample - loss: 0.0680 - mean_absolute_percentage_error: 43.0348 - val_loss: 0.0262 - val_mean_absolute_percentage_error: 20.1073\n",
            "Epoch 6/100\n",
            "35/35 [==============================] - 0s 745us/sample - loss: 0.0402 - mean_absolute_percentage_error: 41.6203 - val_loss: 0.0416 - val_mean_absolute_percentage_error: 29.3647\n",
            "Epoch 7/100\n",
            "35/35 [==============================] - 0s 732us/sample - loss: 0.0441 - mean_absolute_percentage_error: 36.3268 - val_loss: 0.0667 - val_mean_absolute_percentage_error: 33.7147\n",
            "Epoch 8/100\n",
            "35/35 [==============================] - 0s 786us/sample - loss: 0.0507 - mean_absolute_percentage_error: 34.8555 - val_loss: 0.0582 - val_mean_absolute_percentage_error: 31.9228\n",
            "Epoch 9/100\n",
            "35/35 [==============================] - 0s 859us/sample - loss: 0.0435 - mean_absolute_percentage_error: 36.0092 - val_loss: 0.0379 - val_mean_absolute_percentage_error: 28.8918\n",
            "Epoch 10/100\n",
            "35/35 [==============================] - 0s 747us/sample - loss: 0.0383 - mean_absolute_percentage_error: 39.8574 - val_loss: 0.0293 - val_mean_absolute_percentage_error: 23.9395\n",
            "Epoch 11/100\n",
            "35/35 [==============================] - 0s 764us/sample - loss: 0.0436 - mean_absolute_percentage_error: 44.1764 - val_loss: 0.0290 - val_mean_absolute_percentage_error: 20.0000\n",
            "Epoch 12/100\n",
            "35/35 [==============================] - 0s 823us/sample - loss: 0.0420 - mean_absolute_percentage_error: 36.9044 - val_loss: 0.0303 - val_mean_absolute_percentage_error: 23.3255\n",
            "Epoch 13/100\n",
            "35/35 [==============================] - 0s 831us/sample - loss: 0.0380 - mean_absolute_percentage_error: 39.9483 - val_loss: 0.0337 - val_mean_absolute_percentage_error: 26.4391\n",
            "Epoch 14/100\n",
            "35/35 [==============================] - 0s 918us/sample - loss: 0.0383 - mean_absolute_percentage_error: 38.9423 - val_loss: 0.0384 - val_mean_absolute_percentage_error: 27.6047\n",
            "Epoch 15/100\n",
            "35/35 [==============================] - 0s 826us/sample - loss: 0.0380 - mean_absolute_percentage_error: 35.2859 - val_loss: 0.0423 - val_mean_absolute_percentage_error: 26.1917\n",
            "Epoch 16/100\n",
            "35/35 [==============================] - 0s 724us/sample - loss: 0.0394 - mean_absolute_percentage_error: 33.7857 - val_loss: 0.0404 - val_mean_absolute_percentage_error: 24.4440\n",
            "Epoch 17/100\n",
            "35/35 [==============================] - 0s 671us/sample - loss: 0.0391 - mean_absolute_percentage_error: 33.6447 - val_loss: 0.0368 - val_mean_absolute_percentage_error: 24.8876\n",
            "Epoch 18/100\n",
            "35/35 [==============================] - 0s 752us/sample - loss: 0.0390 - mean_absolute_percentage_error: 39.0913 - val_loss: 0.0306 - val_mean_absolute_percentage_error: 25.0254\n",
            "Epoch 19/100\n",
            "35/35 [==============================] - 0s 808us/sample - loss: 0.0397 - mean_absolute_percentage_error: 40.4396 - val_loss: 0.0316 - val_mean_absolute_percentage_error: 24.8026\n",
            "Epoch 20/100\n",
            "35/35 [==============================] - 0s 781us/sample - loss: 0.0383 - mean_absolute_percentage_error: 39.0221 - val_loss: 0.0341 - val_mean_absolute_percentage_error: 26.1505\n",
            "Epoch 21/100\n",
            "35/35 [==============================] - 0s 668us/sample - loss: 0.0381 - mean_absolute_percentage_error: 37.9055 - val_loss: 0.0353 - val_mean_absolute_percentage_error: 25.8629\n",
            "Epoch 22/100\n",
            "35/35 [==============================] - 0s 716us/sample - loss: 0.0385 - mean_absolute_percentage_error: 37.9434 - val_loss: 0.0365 - val_mean_absolute_percentage_error: 24.8363\n",
            "Epoch 23/100\n",
            "35/35 [==============================] - 0s 632us/sample - loss: 0.0384 - mean_absolute_percentage_error: 37.1647 - val_loss: 0.0360 - val_mean_absolute_percentage_error: 25.6325\n",
            "Epoch 24/100\n",
            "35/35 [==============================] - 0s 684us/sample - loss: 0.0382 - mean_absolute_percentage_error: 36.5625 - val_loss: 0.0354 - val_mean_absolute_percentage_error: 26.1762\n",
            "Epoch 25/100\n",
            "35/35 [==============================] - 0s 677us/sample - loss: 0.0384 - mean_absolute_percentage_error: 37.5807 - val_loss: 0.0317 - val_mean_absolute_percentage_error: 24.9922\n",
            "Epoch 26/100\n",
            "35/35 [==============================] - 0s 634us/sample - loss: 0.0392 - mean_absolute_percentage_error: 39.7816 - val_loss: 0.0314 - val_mean_absolute_percentage_error: 23.6788\n",
            "Epoch 27/100\n",
            "35/35 [==============================] - 0s 677us/sample - loss: 0.0382 - mean_absolute_percentage_error: 37.7479 - val_loss: 0.0336 - val_mean_absolute_percentage_error: 25.8214\n",
            "Epoch 28/100\n",
            "35/35 [==============================] - 0s 635us/sample - loss: 0.0396 - mean_absolute_percentage_error: 39.1087 - val_loss: 0.0391 - val_mean_absolute_percentage_error: 28.0559\n",
            "Epoch 29/100\n",
            "35/35 [==============================] - 0s 681us/sample - loss: 0.0400 - mean_absolute_percentage_error: 32.6750 - val_loss: 0.0426 - val_mean_absolute_percentage_error: 24.6492\n",
            "Epoch 30/100\n",
            "35/35 [==============================] - 0s 696us/sample - loss: 0.0404 - mean_absolute_percentage_error: 35.4442 - val_loss: 0.0345 - val_mean_absolute_percentage_error: 25.2430\n",
            "Epoch 31/100\n",
            "35/35 [==============================] - 0s 799us/sample - loss: 0.0399 - mean_absolute_percentage_error: 42.5052 - val_loss: 0.0350 - val_mean_absolute_percentage_error: 26.3707\n",
            "Epoch 32/100\n",
            "35/35 [==============================] - 0s 674us/sample - loss: 0.0394 - mean_absolute_percentage_error: 32.0588 - val_loss: 0.0482 - val_mean_absolute_percentage_error: 26.7856\n",
            "Epoch 33/100\n",
            "35/35 [==============================] - 0s 671us/sample - loss: 0.0416 - mean_absolute_percentage_error: 36.8677 - val_loss: 0.0370 - val_mean_absolute_percentage_error: 26.2617\n",
            "Epoch 34/100\n",
            "35/35 [==============================] - 0s 655us/sample - loss: 0.0389 - mean_absolute_percentage_error: 36.6695 - val_loss: 0.0351 - val_mean_absolute_percentage_error: 25.8119\n",
            "Epoch 35/100\n",
            "35/35 [==============================] - 0s 644us/sample - loss: 0.0381 - mean_absolute_percentage_error: 37.4172 - val_loss: 0.0343 - val_mean_absolute_percentage_error: 26.7241\n",
            "Epoch 36/100\n",
            "35/35 [==============================] - 0s 736us/sample - loss: 0.0388 - mean_absolute_percentage_error: 40.5537 - val_loss: 0.0324 - val_mean_absolute_percentage_error: 26.5931\n",
            "Epoch 37/100\n",
            "35/35 [==============================] - 0s 678us/sample - loss: 0.0384 - mean_absolute_percentage_error: 39.3858 - val_loss: 0.0346 - val_mean_absolute_percentage_error: 24.6551\n",
            "Epoch 38/100\n",
            "35/35 [==============================] - 0s 687us/sample - loss: 0.0382 - mean_absolute_percentage_error: 37.2376 - val_loss: 0.0393 - val_mean_absolute_percentage_error: 24.0049\n",
            "Epoch 39/100\n",
            "35/35 [==============================] - 0s 678us/sample - loss: 0.0412 - mean_absolute_percentage_error: 32.9674 - val_loss: 0.0370 - val_mean_absolute_percentage_error: 24.2000\n",
            "Epoch 40/100\n",
            "35/35 [==============================] - 0s 664us/sample - loss: 0.0391 - mean_absolute_percentage_error: 41.1487 - val_loss: 0.0304 - val_mean_absolute_percentage_error: 25.3624\n",
            "Epoch 41/100\n",
            "35/35 [==============================] - 0s 743us/sample - loss: 0.0398 - mean_absolute_percentage_error: 42.0001 - val_loss: 0.0343 - val_mean_absolute_percentage_error: 25.7569\n",
            "Epoch 42/100\n",
            "35/35 [==============================] - 0s 778us/sample - loss: 0.0381 - mean_absolute_percentage_error: 38.6668 - val_loss: 0.0376 - val_mean_absolute_percentage_error: 27.1323\n",
            "Epoch 43/100\n",
            "35/35 [==============================] - 0s 698us/sample - loss: 0.0382 - mean_absolute_percentage_error: 37.3946 - val_loss: 0.0403 - val_mean_absolute_percentage_error: 26.5623\n",
            "Epoch 44/100\n",
            "35/35 [==============================] - 0s 651us/sample - loss: 0.0393 - mean_absolute_percentage_error: 33.6592 - val_loss: 0.0365 - val_mean_absolute_percentage_error: 24.2606\n",
            "Epoch 45/100\n",
            "35/35 [==============================] - 0s 645us/sample - loss: 0.0396 - mean_absolute_percentage_error: 35.6431 - val_loss: 0.0296 - val_mean_absolute_percentage_error: 24.2568\n",
            "Epoch 46/100\n",
            "35/35 [==============================] - 0s 696us/sample - loss: 0.0390 - mean_absolute_percentage_error: 40.5911 - val_loss: 0.0349 - val_mean_absolute_percentage_error: 26.9537\n",
            "Epoch 47/100\n",
            "35/35 [==============================] - 0s 900us/sample - loss: 0.0397 - mean_absolute_percentage_error: 38.6967 - val_loss: 0.0381 - val_mean_absolute_percentage_error: 27.8834\n",
            "Epoch 48/100\n",
            "35/35 [==============================] - 0s 695us/sample - loss: 0.0403 - mean_absolute_percentage_error: 40.7488 - val_loss: 0.0383 - val_mean_absolute_percentage_error: 23.7432\n",
            "Epoch 49/100\n",
            "35/35 [==============================] - 0s 717us/sample - loss: 0.0390 - mean_absolute_percentage_error: 34.6170 - val_loss: 0.0366 - val_mean_absolute_percentage_error: 25.7099\n",
            "Epoch 50/100\n",
            "35/35 [==============================] - 0s 870us/sample - loss: 0.0411 - mean_absolute_percentage_error: 37.6178 - val_loss: 0.0330 - val_mean_absolute_percentage_error: 27.1938\n",
            "Epoch 51/100\n",
            "35/35 [==============================] - 0s 756us/sample - loss: 0.0387 - mean_absolute_percentage_error: 40.9319 - val_loss: 0.0375 - val_mean_absolute_percentage_error: 25.5870\n",
            "Epoch 52/100\n",
            "35/35 [==============================] - 0s 750us/sample - loss: 0.0418 - mean_absolute_percentage_error: 39.4853 - val_loss: 0.0421 - val_mean_absolute_percentage_error: 24.4247\n",
            "Epoch 53/100\n",
            "35/35 [==============================] - 0s 788us/sample - loss: 0.0387 - mean_absolute_percentage_error: 41.1079 - val_loss: 0.0347 - val_mean_absolute_percentage_error: 27.6516\n",
            "Epoch 54/100\n",
            "35/35 [==============================] - 0s 816us/sample - loss: 0.0417 - mean_absolute_percentage_error: 45.7743 - val_loss: 0.0307 - val_mean_absolute_percentage_error: 25.7710\n",
            "Epoch 55/100\n",
            "35/35 [==============================] - 0s 719us/sample - loss: 0.0389 - mean_absolute_percentage_error: 40.6163 - val_loss: 0.0331 - val_mean_absolute_percentage_error: 22.6059\n",
            "Epoch 56/100\n",
            "35/35 [==============================] - 0s 796us/sample - loss: 0.0419 - mean_absolute_percentage_error: 33.3361 - val_loss: 0.0394 - val_mean_absolute_percentage_error: 24.5434\n",
            "Epoch 57/100\n",
            "35/35 [==============================] - 0s 675us/sample - loss: 0.0379 - mean_absolute_percentage_error: 32.3632 - val_loss: 0.0346 - val_mean_absolute_percentage_error: 28.2802\n",
            "Epoch 58/100\n",
            "35/35 [==============================] - 0s 975us/sample - loss: 0.0415 - mean_absolute_percentage_error: 42.8242 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 26.0543\n",
            "Epoch 59/100\n",
            "35/35 [==============================] - 0s 669us/sample - loss: 0.0396 - mean_absolute_percentage_error: 35.1262 - val_loss: 0.0464 - val_mean_absolute_percentage_error: 23.1342\n",
            "Epoch 60/100\n",
            "35/35 [==============================] - 0s 757us/sample - loss: 0.0410 - mean_absolute_percentage_error: 33.7063 - val_loss: 0.0423 - val_mean_absolute_percentage_error: 27.7429\n",
            "Epoch 61/100\n",
            "35/35 [==============================] - 0s 805us/sample - loss: 0.0396 - mean_absolute_percentage_error: 40.5701 - val_loss: 0.0361 - val_mean_absolute_percentage_error: 28.5229\n",
            "Epoch 62/100\n",
            "35/35 [==============================] - 0s 1ms/sample - loss: 0.0424 - mean_absolute_percentage_error: 41.5852 - val_loss: 0.0309 - val_mean_absolute_percentage_error: 22.5578\n",
            "Epoch 63/100\n",
            "35/35 [==============================] - 0s 674us/sample - loss: 0.0386 - mean_absolute_percentage_error: 37.2645 - val_loss: 0.0360 - val_mean_absolute_percentage_error: 24.8106\n",
            "Epoch 64/100\n",
            "35/35 [==============================] - 0s 716us/sample - loss: 0.0409 - mean_absolute_percentage_error: 36.2904 - val_loss: 0.0455 - val_mean_absolute_percentage_error: 29.0923\n",
            "Epoch 65/100\n",
            "35/35 [==============================] - 0s 731us/sample - loss: 0.0432 - mean_absolute_percentage_error: 43.8714 - val_loss: 0.0334 - val_mean_absolute_percentage_error: 27.6204\n",
            "Epoch 66/100\n",
            "35/35 [==============================] - 0s 649us/sample - loss: 0.0422 - mean_absolute_percentage_error: 47.8032 - val_loss: 0.0334 - val_mean_absolute_percentage_error: 20.9902\n",
            "Epoch 67/100\n",
            "35/35 [==============================] - 0s 906us/sample - loss: 0.0415 - mean_absolute_percentage_error: 36.2853 - val_loss: 0.0326 - val_mean_absolute_percentage_error: 24.4588\n",
            "Epoch 68/100\n",
            "35/35 [==============================] - 0s 678us/sample - loss: 0.0371 - mean_absolute_percentage_error: 37.7962 - val_loss: 0.0389 - val_mean_absolute_percentage_error: 28.5098\n",
            "Epoch 69/100\n",
            "35/35 [==============================] - 0s 675us/sample - loss: 0.0395 - mean_absolute_percentage_error: 38.2019 - val_loss: 0.0427 - val_mean_absolute_percentage_error: 28.8511\n",
            "Epoch 70/100\n",
            "35/35 [==============================] - 0s 673us/sample - loss: 0.0399 - mean_absolute_percentage_error: 36.1976 - val_loss: 0.0415 - val_mean_absolute_percentage_error: 26.5284\n",
            "Epoch 71/100\n",
            "35/35 [==============================] - 0s 663us/sample - loss: 0.0378 - mean_absolute_percentage_error: 34.8055 - val_loss: 0.0316 - val_mean_absolute_percentage_error: 23.6727\n",
            "Epoch 72/100\n",
            "35/35 [==============================] - 0s 641us/sample - loss: 0.0395 - mean_absolute_percentage_error: 38.6054 - val_loss: 0.0296 - val_mean_absolute_percentage_error: 23.1205\n",
            "Epoch 73/100\n",
            "35/35 [==============================] - 0s 658us/sample - loss: 0.0400 - mean_absolute_percentage_error: 42.1685 - val_loss: 0.0361 - val_mean_absolute_percentage_error: 27.0867\n",
            "Epoch 74/100\n",
            "35/35 [==============================] - 0s 674us/sample - loss: 0.0380 - mean_absolute_percentage_error: 37.8014 - val_loss: 0.0447 - val_mean_absolute_percentage_error: 25.8171\n",
            "Epoch 75/100\n",
            "35/35 [==============================] - 0s 691us/sample - loss: 0.0409 - mean_absolute_percentage_error: 33.3442 - val_loss: 0.0449 - val_mean_absolute_percentage_error: 26.9463\n",
            "Epoch 76/100\n",
            "35/35 [==============================] - 0s 956us/sample - loss: 0.0389 - mean_absolute_percentage_error: 34.4400 - val_loss: 0.0324 - val_mean_absolute_percentage_error: 24.4327\n",
            "Epoch 77/100\n",
            "35/35 [==============================] - 0s 678us/sample - loss: 0.0386 - mean_absolute_percentage_error: 36.9492 - val_loss: 0.0308 - val_mean_absolute_percentage_error: 25.4985\n",
            "Epoch 78/100\n",
            "35/35 [==============================] - 0s 670us/sample - loss: 0.0396 - mean_absolute_percentage_error: 39.2027 - val_loss: 0.0301 - val_mean_absolute_percentage_error: 23.4169\n",
            "Epoch 79/100\n",
            "35/35 [==============================] - 0s 705us/sample - loss: 0.0389 - mean_absolute_percentage_error: 38.8427 - val_loss: 0.0322 - val_mean_absolute_percentage_error: 24.2110\n",
            "Epoch 80/100\n",
            "35/35 [==============================] - 0s 743us/sample - loss: 0.0378 - mean_absolute_percentage_error: 37.4581 - val_loss: 0.0378 - val_mean_absolute_percentage_error: 26.6263\n",
            "Epoch 81/100\n",
            "35/35 [==============================] - 0s 707us/sample - loss: 0.0388 - mean_absolute_percentage_error: 35.4535 - val_loss: 0.0455 - val_mean_absolute_percentage_error: 28.0387\n",
            "Epoch 82/100\n",
            "35/35 [==============================] - 0s 650us/sample - loss: 0.0397 - mean_absolute_percentage_error: 36.1303 - val_loss: 0.0357 - val_mean_absolute_percentage_error: 25.1957\n",
            "Epoch 83/100\n",
            "35/35 [==============================] - 0s 811us/sample - loss: 0.0385 - mean_absolute_percentage_error: 38.7578 - val_loss: 0.0314 - val_mean_absolute_percentage_error: 24.3773\n",
            "Epoch 84/100\n",
            "35/35 [==============================] - 0s 728us/sample - loss: 0.0384 - mean_absolute_percentage_error: 37.5197 - val_loss: 0.0352 - val_mean_absolute_percentage_error: 25.3775\n",
            "Epoch 85/100\n",
            "35/35 [==============================] - 0s 679us/sample - loss: 0.0394 - mean_absolute_percentage_error: 37.3282 - val_loss: 0.0381 - val_mean_absolute_percentage_error: 27.2179\n",
            "Epoch 86/100\n",
            "35/35 [==============================] - 0s 667us/sample - loss: 0.0381 - mean_absolute_percentage_error: 36.6569 - val_loss: 0.0329 - val_mean_absolute_percentage_error: 25.2801\n",
            "Epoch 87/100\n",
            "35/35 [==============================] - 0s 718us/sample - loss: 0.0391 - mean_absolute_percentage_error: 40.6502 - val_loss: 0.0279 - val_mean_absolute_percentage_error: 22.8830\n",
            "Epoch 88/100\n",
            "35/35 [==============================] - 0s 837us/sample - loss: 0.0399 - mean_absolute_percentage_error: 40.3445 - val_loss: 0.0337 - val_mean_absolute_percentage_error: 24.7839\n",
            "Epoch 89/100\n",
            "35/35 [==============================] - 0s 677us/sample - loss: 0.0382 - mean_absolute_percentage_error: 37.4946 - val_loss: 0.0481 - val_mean_absolute_percentage_error: 28.2170\n",
            "Epoch 90/100\n",
            "35/35 [==============================] - 0s 720us/sample - loss: 0.0400 - mean_absolute_percentage_error: 36.0338 - val_loss: 0.0419 - val_mean_absolute_percentage_error: 27.3621\n",
            "Epoch 91/100\n",
            "35/35 [==============================] - 0s 617us/sample - loss: 0.0391 - mean_absolute_percentage_error: 35.8857 - val_loss: 0.0324 - val_mean_absolute_percentage_error: 23.9461\n",
            "Epoch 92/100\n",
            "35/35 [==============================] - 0s 701us/sample - loss: 0.0397 - mean_absolute_percentage_error: 40.2459 - val_loss: 0.0304 - val_mean_absolute_percentage_error: 23.9141\n",
            "Epoch 93/100\n",
            "35/35 [==============================] - 0s 626us/sample - loss: 0.0398 - mean_absolute_percentage_error: 38.5679 - val_loss: 0.0395 - val_mean_absolute_percentage_error: 26.4940\n",
            "Epoch 94/100\n",
            "35/35 [==============================] - 0s 748us/sample - loss: 0.0391 - mean_absolute_percentage_error: 34.6138 - val_loss: 0.0363 - val_mean_absolute_percentage_error: 24.9717\n",
            "Epoch 95/100\n",
            "35/35 [==============================] - 0s 692us/sample - loss: 0.0382 - mean_absolute_percentage_error: 35.4741 - val_loss: 0.0342 - val_mean_absolute_percentage_error: 26.1449\n",
            "Epoch 96/100\n",
            "35/35 [==============================] - 0s 662us/sample - loss: 0.0393 - mean_absolute_percentage_error: 39.5697 - val_loss: 0.0320 - val_mean_absolute_percentage_error: 24.9750\n",
            "Epoch 97/100\n",
            "35/35 [==============================] - 0s 644us/sample - loss: 0.0395 - mean_absolute_percentage_error: 38.5554 - val_loss: 0.0372 - val_mean_absolute_percentage_error: 26.9449\n",
            "Epoch 98/100\n",
            "35/35 [==============================] - 0s 675us/sample - loss: 0.0430 - mean_absolute_percentage_error: 37.0881 - val_loss: 0.0346 - val_mean_absolute_percentage_error: 22.9008\n",
            "Epoch 99/100\n",
            "35/35 [==============================] - 0s 715us/sample - loss: 0.0422 - mean_absolute_percentage_error: 34.1944 - val_loss: 0.0338 - val_mean_absolute_percentage_error: 27.3413\n",
            "Epoch 100/100\n",
            "35/35 [==============================] - 0s 711us/sample - loss: 0.0389 - mean_absolute_percentage_error: 39.3782 - val_loss: 0.0333 - val_mean_absolute_percentage_error: 24.8787\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f9fad2f7a58>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pNB5k_MIoPGv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Predictions\n",
        "pred_train = model.predict(standardized_X_train)\n",
        "pred_test = model.predict(standardized_X_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f8jVRQhuoPEI",
        "colab_type": "code",
        "outputId": "591d72a8-8f52-4b3d-cfa8-c6bba55f04ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Train and test MSE\n",
        "train_mse = np.mean((standardized_y_train - pred_train) ** 2)\n",
        "test_mse = np.mean((standardized_y_test - pred_test) ** 2)\n",
        "print (f\"train_MSE: {train_mse:.2f}, test_MSE: {test_mse:.2f}\")"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train_MSE: 0.03, test_MSE: 2.24\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-4X3GZEdoR-8",
        "colab_type": "code",
        "outputId": "eea237a4-a2b3-4d40-f24f-6c64082fda37",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Unstandardize coefficients (proof is in the `From Scratch` section above)\n",
        "W = model.layers[0].get_weights()[0][0][0]\n",
        "b = model.layers[0].get_weights()[1][0]\n",
        "W_unscaled = W * (y_scaler.scale_/X_scaler.scale_)\n",
        "b_unscaled = b * y_scaler.scale_ + y_scaler.mean_ - np.sum(W_unscaled*X_scaler.mean_)\n",
        "print (\"[actual] y = 3.5X + noise\")\n",
        "print (f\"[model] y_hat = {W_unscaled[0]:.1f}X + {b_unscaled[0]:.1f}\") "
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[actual] y = 3.5X + noise\n",
            "[model] y_hat = 3.4X + 9.5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mdNX2W5eh2ma",
        "colab_type": "text"
      },
      "source": [
        "Regularization didn't help much with this specific example because our data is generated from a perfect linear equation but for large realistic data, regularization can help our model generalize well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V74lNFE5v5pQ",
        "colab_type": "text"
      },
      "source": [
        "# Categorical variables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2r6Xhyg7v5vX",
        "colab_type": "text"
      },
      "source": [
        "In our example, the feature was a continuous variable but what if we also have features that are categorical? One option is to treat the categorical variables as one-hot encoded variables. This is very easy to do with Pandas and once you create the dummy variables, you can use the same steps as above to train your linear model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "unhcIOfMxQEQ",
        "colab_type": "code",
        "outputId": "458bfea8-d946-4cd6-cf1c-5943a952008c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "source": [
        "# Create data with categorical features\n",
        "cat_data = pd.DataFrame(['a', 'b', 'c', 'a'], columns=['favorite_letter'])\n",
        "cat_data.head()"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>favorite_letter</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>b</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>c</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "  favorite_letter\n",
              "0               a\n",
              "1               b\n",
              "2               c\n",
              "3               a"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m4eQmJdrxQGr",
        "colab_type": "code",
        "outputId": "b6bf3b36-c210-4587-b73a-617c9da6f6ed",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "source": [
        "dummy_cat_data = pd.get_dummies(cat_data)\n",
        "dummy_cat_data.head()"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>favorite_letter_a</th>\n",
              "      <th>favorite_letter_b</th>\n",
              "      <th>favorite_letter_c</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   favorite_letter_a  favorite_letter_b  favorite_letter_c\n",
              "0                  1                  0                  0\n",
              "1                  0                  1                  0\n",
              "2                  0                  0                  1\n",
              "3                  1                  0                  0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5R8x-KyiBWJ",
        "colab_type": "text"
      },
      "source": [
        "Now you can concat this with your continuous features and train the linear model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XwSZgi_pCb3Q",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "<div align=\"center\">\n",
        "\n",
        "Subscribe to our <a href=\"https://practicalai.me/#newsletter\">newsletter</a> and follow us on social media to get the latest updates!\n",
        "\n",
        "<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://github.com/practicalAI/practicalAI\">\n",
        "              <img src=\"https://img.shields.io/github/stars/practicalAI/practicalAI.svg?style=social&label=Star\"></a>&nbsp;\n",
        "            <a class=\"ai-header-badge\" target=\"_blank\" href=\"https://www.linkedin.com/company/madewithml\">\n",
        "              <img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n",
        "            <a class=\"ai-header-badge\" target=\"_blank\" href=\"https://twitter.com/madewithml\">\n",
        "              <img src=\"https://img.shields.io/twitter/follow/madewithml.svg?label=Follow&style=social\">\n",
        "            </a>\n",
        "              </div>\n",
        "\n",
        "</div>"
      ]
    }
  ]
}
